{"id":"9762e878-0b12-457c-84fb-5f278aec178f","arxiv_id":"2501.15145","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"PromptShield reports a 65.3% true positive rate at 0.1% false positive rate for prompt injection detection, more than six times the best prior model, on its own out-of-distribution evaluation split.","lead":"PromptShield introduces a curated benchmark for prompt injection detection and a fine-tuned detector that reports much higher detection rates at very low false positive rates than prior models. The paper is worth reading because it targets the false-alarm problem that has kept such detectors out of production, though its headline numbers come with a methodological caveat.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 65.3% TPR at 0.1% FPR is computed with thresholds selected on the evaluation split itself, not on held-out calibration data; this is admitted in Section 3.2.2 and makes the central deployable claim in-sample.","rationale":"The reader's stated weakest assumption is the benchmark taxonomy that labels all conversational data as benign and excludes multi-turn and function-calling scenarios. That is a real external-validity concern: if real deployments include indirect prompt injections in chatbot contexts or function-calling interfaces, the low-FPR results may not transfer. However, the more load-bearing concern for the paper's central quantitative claim is internal: the headline 65.3% TPR at 0.1% FPR is obtained by selecting the decision threshold on the evaluation split itself. This is not a subtle interpretation issue; it is an explicit, self-identified methodological flaw in Section 3.2.2. A deployable detector must have its threshold set before deployment, typically using a validation set or other calibration data, and evaluated on unseen data. Because the paper selects thresholds on the evaluation split, the reported 0.1% FPR is enforced rather than measured, and the TPR at that threshold is likely optimistic. The problem is amplified at very low FPRs by the small validation set and the acknowledged variability across train/validation splits. This concern does not invalidate the benchmark itself, the released code and data, or the relative ordering of detectors under a common (if flawed) protocol; those are genuine contributions. But it directly undermines the absolute 'deployable' claim in the abstract and introduction. The reader's verdict of CONDITIONAL already captures the need for a corrected evaluation, so I do not recommend changing the verdict; the condition should explicitly require threshold calibration on a held-out validation set and reporting of error bars at low FPR targets.","tokens_in":18333,"tokens_out":2451,"duration_ms":24055,"concrete_test":"Re-run the evaluation using the held-out validation split (or a newly created held-out set of comparable size) to select thresholds with the same bisection procedure at target FPRs of 1%, 0.5%, 0.1%, and 0.05%, then evaluate the so-calibrated model on the evaluation split. Report TPR and achieved FPR on the evaluation split, together with bootstrap confidence intervals over evaluation samples and over multiple validation-split draws. If the Llama-3.1-8B model's TPR at achieved 0.1% FPR is materially below 65.3% (for example, by more than 5 percentage points) or has a wide confidence interval, then the central deployability claim is not supported by the current evaluation protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest claim is that the PromptShield detector achieves 65.3% TPR at 0.1% FPR on its evaluation split (Section 1, Table 4). The deployment scheme in Section 3.2.2, however, caches scores on the evaluation split and selects the decision threshold by interpolating the ROC curve on that same split, then reports TPR at that enforced FPR. This means the quoted 0.1% FPR is true by construction on the evaluation split, and the TPR at that threshold is an in-sample quantity, not an out-of-sample prediction of deployment performance. The paper explicitly acknowledges this: 'Note that in real-life deployment settings model maintainers will not necessarily have access to test data. In retrospect, we should have used the validation split for this calibration step.' The problem is especially severe at 0.1% FPR because the held-out validation set described in Section 3.2.1 contains only about 1000 datapoints; selecting a threshold for a 0.1% FPR requires estimating a quantile near one false positive, so the calibrated threshold will be extremely noisy. Appendix A.3 also concedes that the train/validation split choice causes variation at very low FPR levels. The relative comparison to PromptGuard is less affected because competitor thresholds are also calibrated on the evaluation split, and the AUC numbers are threshold-free. But the headline 'deployable' claim rests on the absolute TPR at 0.1% FPR, and that number has not been shown to transfer to any honest deployment protocol.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PromptShield, a benchmark and a fine-tuned detector for prompt injection attacks in LLM-integrated applications. It defines a deployment-oriented taxonomy separating conversational data from application-structured data, curates benign and attack samples from public datasets and published attack strategies, and provides non-overlapping train and evaluation splits. The PromptShield detector is fine-tuned from several base models, and the authors report AUC and TPR at low target FPRs using a threshold-calibration procedure. The central empirical claim is that the Llama-3.1-8B variant detects 65.3% of attacks at 0.1% FPR on the evaluation split, far outperforming prior detectors such as PromptGuard (9.4%). The paper also includes ablations on model size, training set size, and training data composition, plus a limitations section.","tokens_in":18661,"tokens_out":7236,"duration_ms":63699,"significance":"If the reported performance holds out of sample, the paper makes a useful contribution: it provides a public, reusable benchmark with an explicitly deployment-oriented taxonomy, emphasizes the low-FPR regime that is relevant to practical deployment, and shows a substantial improvement over existing detectors. The curation effort is significant, and the release of the benchmark and code increases the value of the work. At the same time, the headline quantitative claim is currently an in-sample quantity because thresholds are calibrated on the evaluation split, and the single-run results carry no uncertainty estimates. The benchmark contribution is likely to be useful regardless of the exact detector numbers, but the 'deployable' conclusion requires the evaluation protocol to be fixed or carefully qualified.","major_comments":[{"comment":"The headline result in Section 1 and Table 4 is obtained by selecting decision thresholds on the evaluation split itself. Section 3.2.2 states that the authors \"cache model output scores on the evaluation split\" and interpolate the ROC curve on that split to choose thresholds, and it acknowledges that the validation split should have been used. This makes the reported 65.33% TPR at 0.1% FPR an in-sample fitted quantity: the FPR is enforced by construction, and the TPR at that threshold has no demonstrated out-of-sample counterpart. The problem is especially severe at 0.1% FPR because the validation split described in Section 3.2.1 has only about 1000 points, so the targeted FPR corresponds to roughly one false positive. Please recalibrate thresholds on a held-out split and report the resulting TPR, or report both in-sample and held-out values. The AUC numbers in Table 4 and the relative ordering of detectors are less affected, but the absolute deployability claim is not yet supported. Appendix A.3 already notes that the validation-split choice causes variation at very low FPR, which further cautions against treating the point estimate as stable.","section":"Section 3.2.2 and Table 4"},{"comment":"All results are single runs without confidence intervals or variance estimates. Appendix A.3 reports that the choice of training/validation split changes performance at extremely low FPR levels, and Tables 5 and 6 contain several differences of only a few percentage points in this regime (for example, FLAN-T5-base versus Llama-3.2-1B at 0.05% FPR, or the 5K versus 10K training-set comparison at 0.05% FPR). Without multiple seeds or split resamples, the reader cannot determine whether these comparisons are meaningful. Please add an uncertainty estimate for at least the primary PromptShield configuration and for the model-size and training-size comparisons.","section":"Appendix A.3, Tables 5 and 6"},{"comment":"The deployability claim is scoped to a specific taxonomy: all conversational traffic is treated as benign, and multi-turn interaction and function calling are explicitly excluded in Section 2.4. The benchmark labels first-turn conversational data from Ultrachat and LMSYS as benign, while real provider-deployed systems must also handle indirect injections embedded in fetched web content, uploaded files, and multi-turn state. The reported low FPR may therefore not transfer to deployments that include these out-of-scope categories. This is a defensible scoping decision, but the paper should either evaluate on a small sample of such traffic or explicitly restrict the \"deployable\" conclusion to the single-turn, text-only, two-category setting considered in the benchmark.","section":"Sections 2.3, 2.4, and 7"}],"minor_comments":[{"comment":"The sentence \"they report a TPR of 71% and FPR of 1% [7]\" cites the Llama 3 herd technical report; the intended source appears to be the CYBERSECEVAL 3 report [30]. Please correct the citation.","section":"Section 5.1"},{"comment":"There is a typo in \"plateaues\"; it should read \"plateaus.\"","section":"Section 4"},{"comment":"The fact that PromptGuard's \"jailbreak\" class is used as a proxy for prompt injection is important for interpreting Table 4; consider noting this caveat in Section 5.1 where PromptGuard is first discussed.","section":"Appendix B.2"},{"comment":"The caption and table for the conversational-only evaluation leave the AUC column blank; add a short explanation (for example, that ROC analysis cannot be computed on a single-class set) in the caption.","section":"Table 9"},{"comment":"The footnote \"there does not exist a threshold that achieves the desired FPR aside from 1.0\" is ambiguous; clarify whether the issue is tied score values or some other property of the score distribution.","section":"Table 4 footnote"}],"recommendation":"major_revision","confidential_remarks":"I support a major revision rather than rejection because the benchmark contribution is valuable and the calibration flaw is explicitly acknowledged and fixable. The main risk is that out-of-sample results at 0.1% FPR will be substantially weaker; if so, the paper should still be publishable as a benchmark paper, with the detector results presented more cautiously. The claim that the deployment scheme is novel may also need tempering, since target-FPR calibration is common in security evaluation. The authors' candidness about the threshold-calibration limitation is commendable and should be credited in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The PromptShield benchmark is a real asset: it assembles open datasets into a train/eval split that forces out-of-distribution generalization, uses disjoint link phrases for injections, and is publicly released. The detector paper is also unusually honest: it admits in Section 3.2.2 that decision thresholds were calibrated on the evaluation split itself, so the headline 65.3% TPR at 0.1% FPR is fitted on the test set, not an out-of-sample prediction. That is the main thing to tell anyone who quotes that number.\n\nWhat is new and good: the benchmark design is thoughtful. The split is constructed so that detectors are fine-tuned on one distribution and tested on a different one, with attack templates using link phrases unseen at train time. The low-FPR evaluation protocol is the right emphasis for deployed detectors, and the ablation separating conversational data from application-structured data is informative. The paper also ships code and data, which makes the claims checkable. The relative comparison to PromptGuard, ProtectAI, InjecGuard, and Fmops is less affected by the threshold issue because the same calibration method is applied to all competitors, and the AUC numbers are threshold-free.\n\nWhere it is soft: the absolute deployable claim is in-sample. The validation split described in Section 3.2.1 has only about 1,000 points; estimating a 0.1% FPR threshold from that is extremely noisy, and Appendix A.3 concedes split choice causes variation at extremely low FPR. The paper even says in retrospect it should have used the validation split. So the 65.3% number should not be quoted as expected deployment performance until re-evaluated with a validation-set threshold and confidence intervals. Also, the taxonomy is limited to single-turn conversational and application-structured data; multi-turn and function calling are out of scope, so the \"deployable\" framing overreaches a bit. The \"first such benchmark\" claim is arguable given prior benchmarks like InjecGuard and OpenPromptInjection, but the disjoint-link-phrase split is a genuinely new feature.\n\nBottom line: this is a solid empirical contribution with a load-bearing flaw in the headline number, but the flaw is acknowledged and fixable. The benchmark is worth having. I would send it to review, asking for a redo of the threshold protocol and error bars. It belongs in a security venue or an LLM-systems workshop. I would cite the benchmark in my own work but not the absolute TPR claim.","headline":"A useful, publicly released benchmark and an honest paper whose headline detection number is in-sample until the threshold is re-calibrated on held-out data.","tokens_in":19181,"tokens_out":2501,"would_cite":true,"duration_ms":21863,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A detector that catches 65% of prompt injections while flagging only 0.1% of benign traffic.","keywords":["prompt injection","prompt injection detection","large language models","benchmark","fine-tuning","false positive rate","low-FPR deployment","LLM security"],"falsifier":"Take the final PromptShield model, choose its threshold to hit 0.1% FPR on a held-out validation set (not the evaluation split), then measure FPR and TPR on a corpus of multi-turn chatbot sessions that include web-browsing or file-upload content where an attacker has embedded an injected instruction; if the empirical FPR exceeds the target by a wide margin or the detector misses those embedded injections, the paper's deployability claim fails.","tokens_in":18146,"feed_emoji":"🛡️","tokens_out":9109,"duration_ms":70331,"temperature":0.7,"pith_summary":"PromptShield is a benchmark and detector built around a deployment-centric view: prompt injection detectors should be judged at the false positive rates that real traffic forces, because benign requests vastly outnumber attacks. The benchmark splits the input space into conversational data (declared always benign) and application-structured data (the only injection surface), and its train/evaluation split is engineered so detectors cannot memorize the evaluation distribution. Fine-tuning Llama 3.1 8B on this benchmark yields a detector that catches 65.3% of prompt injections at 0.1% false positive rate on the evaluation split, while PromptGuard, the strongest prior scheme, catches 9.4% at the same FPR. The paper concludes that careful data curation plus larger base models, not novel architectures, are what make low-FPR deployment feasible.","feed_headline":"Detector catches 65% of prompt injections at 0.1% false-positive rate","feed_subtitle":"A new benchmark and fine-tuned model outdo prior detectors in the low-false-alarm regime where real deployments operate.","key_machinery":"The central mechanism is the PromptShield benchmark itself: a two-category taxonomy (conversational data treated as always benign, application-structured data treated as the only injection surface) instantiated as a curated mix of benign corpora (Ultrachat, LMSYS, Alpaca, databricks-dolly, natural-instructions, SPP, IFEval) and attack corpora (FourAttacks built from StruQ's naive/ignore/completion/combined templates, HackAPrompt, OpenPromptInject). Its train/evaluation split is designed to prevent memorization by using disjoint datasets and disjoint injection link phrases across the two splits. The companion deployment scheme selects a decision threshold from the ROC curve to hit target false positive rates of 1%, 0.5%, 0.1%, and 0.05%, and reports true positive rates at those thresholds; this calibration is what separates the paper's numbers from default-threshold accuracy.","core_discovery":"The paper's central claim is that a prompt injection detector can be highly sensitive and still deployable if it is fine-tuned on a benchmark that mirrors the real distribution of LLM traffic and evaluated at threshold-calibrated low false positive rates. On the PromptShield evaluation split, the fine-tuned Llama-3.1-8B detector achieves 94.8% true positive rate at 1% FPR, 87.8% at 0.5% FPR, 65.3% at 0.1% FPR, and 47.5% at 0.05% FPR, while PromptGuard, the strongest prior scheme, achieves 12.8%, 12.4%, 9.4%, and 1.5% at the same FPRs. Ablations show that enlarging the training set from 1K to 20K samples more than doubles true positive rate at the most stringent FPRs, and that adding conversational training data cuts false positives on chatbot traffic by about 25% at the least strict threshold while a modest TPR cost appears only at the lowest FPRs. The authors conclude that careful curation and larger base models, rather than architectural innovation, drive performance in the deployment-relevant regime.","pith_inferences":["If real chatbots gain web-browsing or file-upload features, an attacker can plant an injection inside content a user requests, which the benchmark's taxonomy would still label as benign conversational data; testing PromptShield on such embedded injections would show whether its low-FPR advantage survives outside the taxonomy.","The paper admits it selected decision thresholds using the evaluation split and recommends a validation split instead; re-running the protocol with a disjoint calibration set is the direct way to measure how much the headline 65.3%-at-0.1% figure is an artifact of evaluation-split threshold tuning.","Since optimization-based attacks such as GCG are excluded from scope, the claimed performance is established only against template-based attacks; an adaptive adversary who optimizes a suffix to flip the detector's output would test whether the low-FPR edge holds under attack."],"forward_implications":["With the deployment scheme's threshold in place, a provider-deployed detector can block roughly two-thirds of prompt injections at 0.1% FPR, meaning fewer than one in a thousand benign requests gets refused.","Reporting TPR at target FPRs instead of AUC exposes that prior detectors (PromptGuard, ProtectAI, InjecGuard, Fmops) are effectively non-functional in the low-FPR regime; the paper suggests low-FPR TPR should become the standard metric.","Because scaling the training set from 1K to 20K samples and the base model from 61M to 8B parameters both substantially raise low-FPR TPR, the results point to data and model scale as the practical levers for detector improvement.","The extendable benchmark construction means newly published attacks and datasets can be added, so future detectors can be trained against a moving attack distribution rather than a static one."],"supporting_citations":[{"why":"StruQ supplies the four template attack strategies (naive, ignore, completion, combined) used to build the benchmark's injection samples.","marker":"[4]"},{"why":"The Llama 3 model family provides the base architecture whose 8B variant, fine-tuned with LoRA, yields the leading PromptShield detector.","marker":"[7]"},{"why":"Ultrachat contributes the conversational benign training corpus drawn from ChatGPT interactions.","marker":"[6]"},{"why":"OpenPromptInject provides additional attack seeds and an injection framework that diversify the benchmark's injection data.","marker":"[14]"},{"why":"HackAPrompt contributes crowd-sourced real-world injections used in the training split.","marker":"[24]"},{"why":"Alpaca is the instruction-following corpus that anchors the application-structured benign data and seeds the FourAttacks injections.","marker":"[26]"},{"why":"PromptGuard is the strongest prior detector and the main baseline, catching only 9.4% of attacks at 0.1% FPR on the evaluation split.","marker":"[30]"},{"why":"LMSYS supplies the real-world chatbot conversation corpus used as evaluation conversational data.","marker":"[37]"}],"fun_headline_variants":["Prompt injection detector tuned for deployment hits 65% recall at 0.1% FPR","Curated data boosts prompt-injection detection to 94.8% TPR at 1% FPR","Larger models and careful data beat architecture in injection defense","New benchmark and fine-tuned model make prompt-injection detection deployable","PromptShield: a benchmark to train deployable prompt-injection detectors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire 'deployable' framing rests on the assumption that real traffic is fully described by two categories—conversational requests, which are always benign, and application-structured data, which is the only place injections appear—with multi-turn chats, function calling, and injections inside conversational content excluded as out of scope.","fun_headline_variants_meta":{"raw":{"variants":["Prompt injection detector tuned for deployment hits 65% recall at 0.1% FPR","Curated data boosts prompt-injection detection to 94.8% TPR at 1% FPR","Larger models and careful data beat architecture in injection defense","New benchmark and fine-tuned model make prompt-injection detection deployable","PromptShield: a benchmark to train deployable prompt-injection detectors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000703,"raw_usage":{"total_tokens":3167,"prompt_tokens":936,"completion_tokens":2231,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":2125}},"tokens_in":552,"tokens_out":2231,"duration_ms":14943,"temperature":1.0,"reasoning_tokens":2125,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:34:30.379369+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the final PromptShield model, choose its threshold to hit 0.1% FPR on a held-out validation set (not the evaluation split), then measure FPR and TPR on a corpus of multi-turn chatbot sessions that include web-browsing or file-upload content where an attacker has embedded an injected instruction; if the empirical FPR exceeds the target by a wide margin or the detector misses those embedded injections, the paper's deployability claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"StruQ supplies the four template attack strategies (naive, ignore, completion, combined) used to build the benchmark's injection samples."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Alpaca is the instruction-following corpus that anchors the application-structured benign data and seeds the FourAttacks injections."}],"review_version":1}