{"id":"c1a52ea2-5077-407f-8000-c9a75336bbe9","arxiv_id":"2505.13028","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"An evaluation of seven LLM security tools on a new 500-prompt benchmark finds the ChatGPT-3.5-Turbo baseline unusable due to false positives and names Lakera Guard and ProtectAI LLM Guard the best overall tools.","lead":"This paper built a benchmark of 500 malicious and benign prompts and tested seven LLM security products plus ChatGPT-3.5-Turbo against it. It finds the baseline model flags too many benign queries, while Lakera Guard and ProtectAI LLM Guard offer the best balance of accuracy, false positives, and speed.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline ranking rests on an admitted, untested representativeness assumption: the benchmark's attack prompts come from public research generators, not threat intelligence, and internal results show rankings vary sharply by prompt source.","rationale":"I read the paper in good faith. It is an honest, carefully reported empirical study: the authors describe their search process, report limitations explicitly, and provide full result tables. The negative claim about ChatGPT-3.5-Turbo's false-positive rate is robust because it appears across all three datasets (FPR 0.431 to 0.859), including the externally sourced Deepset data. The more fragile part of the central claim is the positive ranking of specific tools as 'best overall.' That ranking is not tied to a defined composite metric, and the paper's own ablation shows strong dependence on attack source. The admitted lack of threat-intelligence-based prompts in Section 5.3 means the benchmark's external validity is unestablished, and the failure to release the dataset prevents checking for overlap with vendor training data. These issues are precisely what the reader flagged, so I agree with the weakest-assumption analysis. The proposed held-out test would settle whether the ranking reflects genuine generalization or overfitting to public attack templates. Since the reader already issued a CONDITIONAL verdict and the paper has no fatal internal inconsistency, I recommend no change to that verdict.","tokens_in":17291,"tokens_out":5502,"duration_ms":62406,"concrete_test":"Build a held-out validation set from sources that postdate the tools' release or were never public: for example, a private red-team exercise and a sample of recently collected real-world jailbreak and injection reports, plus benign production queries from a deployed chatbot. Re-run the same seven tools and recompute FPR, recall, ASR by source, and the resulting top-2 ranking. If Lakera and ProtectAI do not remain top two under this set, or if their per-source ASR is much worse on non-public prompts than on Palit prompts, the headline ranking is an artifact of public-prompt overlap and benchmark composition.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central practical claim—that Lakera Guard and ProtectAI LLM Guard are the 'best overall' tools and that ChatGPT-3.5-Turbo is unusable due to false positives—requires the Palit benchmark to be a valid proxy for real-world malicious prompts. Section 5.3 explicitly disclaims this: 'it is unclear whether our attacks are representative of malicious prompts used by real-world attackers, given they were inspired by attacks created by researchers and not threat intelligence.' That disclaimer is not a peripheral caveat; it is the load-bearing condition for the ranking. The risk is concrete, not hypothetical: all attack prompts were produced by manual templates or public generators (Houyi, Garak, PromptMap, Deepset), so commercial tools may have been trained on identical or similar examples, inflating benchmark detection relative to novel attacks. Internal evidence confirms that the ranking is prompt-source-dependent: with context, baseline ASR is 100% on manual prompts but 0% on Garak and PromptMap prompts (Table 7); Lakera's ASR on PromptMap prompts is 83.05% even where its overall ASR is among the lowest. Because the winning set is not invariant across attack sources, 'best overall' is an artifact of the particular 75.8%-malicious mix unless representativeness is established. The benchmark and code are not released, so the source composition and any overlap with vendor training data cannot even be audited.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a comparative evaluation of seven LLM security tools (plus a ChatGPT-3.5-Turbo baseline) against a newly constructed benchmark called Palit, comprising 500 prompts (75.8% malicious) drawn from manual generation, Houyi, Garak, PromptMap, and the public Deepset prompt-injection dataset. The authors report precision, recall, accuracy, FPR, F1, latency, and attack success rates under three conditions: no context, with base-prompt context, and on the Deepset dataset. Their main conclusions are that the ChatGPT-3.5-Turbo baseline is unsuitable due to high false-positive rates (FPR 0.438-0.859), that Lakera Guard and ProtectAI LLM Guard are the best overall tools, and that the market of bolt-on LLM security solutions suffers from limited transparency and usability issues. The paper also surveys 13 tools and compares their claimed features.","tokens_in":17517,"tokens_out":3032,"duration_ms":32684,"significance":"If the results hold, this would be one of the first independent, published evaluations of commercial and open-source LLM security tools, and it would provide a useful template for future benchmarking in this area. The paper's strengths are its timely research question, the explicit attention to usability (latency, deployment, trial availability), the multi-source attack generation strategy, the ablation on system-prompt context, and the remarkably honest limitations section. The data tables support the qualitative claim that the baseline has a very high FPR and that several tools (especially Lakera, ProtectAI, and Azure Prompt Shield) achieve high precision with low FPR. However, the headline 'best overall' ranking is built on an undefined composite metric and on a benchmark whose representativeness of real-world attacks the authors themselves disclaim; the per-source ASR tables show that the ranking shifts substantially depending on prompt source. The paper is therefore a useful empirical contribution whose central claim needs substantial qualification before it can be accepted as stated.","major_comments":[{"comment":"The claim that 'Lakera Guard and ProtectAI LLM Guard emerged as the best overall tools' is not supported by any explicit definition of 'best overall.' The paper reports six separate metrics (accuracy, precision, recall, FPR, F1, latency) and no rule for combining them. Different tools win on different dimensions: e.g., Lakera has the highest accuracy on the benchmark with context (74.6%) and lowest latency (0.305s), but Azure Prompt Shield has higher precision (0.952) and lower FPR (0.074); on the Deepset dataset, Azure achieves precision 1.0 while Lakera achieves higher recall. Without a stated aggregation rule or a decision-theoretic justification (e.g., a cost model for false positives vs. false negatives vs. latency), the ordering is not reproducible and the 'best overall' wording should be replaced with a statement about specific metrics.","section":"Sections 4.3 and 5.2"},{"comment":"The external validity of the Palit benchmark is load-bearing for the ranking, and the manuscript itself disclaims it: 'it is unclear whether our attacks are representative of malicious prompts used by real-world attackers, given they were inspired by attacks created by researchers and not threat intelligence.' The internal results show that the ranking is not invariant across attack sources. For example, with context the baseline ASR is 100% on manual prompts but 0% on Garak and PromptMap prompts (Table 7), and Lakera's ASR on PromptMap prompts is 83.05% even though its overall ASR is low. This means the headline ranking could be an artifact of the particular 75.8%-malicious mix of manual/Houyi/Garak/PromptMap prompts. The authors should either (a) provide evidence that this mix approximates real-world malicious prompt distributions, or (b) restrict their conclusions to per-source results and characterize for which prompt families each tool is effective. Since the dataset is not released (no link or repository is given), readers cannot even check for overlaps with vendor training data, a risk the authors themselves raise for the Deepset portion in Section 5.2.","section":"Section 5.3, Tables 6 and 7"},{"comment":"All reported metrics are point estimates on a single dataset of 500 prompts, with no confidence intervals, bootstrap estimates, or significance tests. This is especially problematic for the per-source ASR tables, where the number of prompts per cell is small (e.g., the Garak-generated subset contains 13 DAN prompts in addition to encoding-based attacks, and the manual subset yields only 59-81 prompts per context condition). Differences that appear large, such as Vigil's drop in ASR from 93.22% to 70.37% on PromptMap prompts when context is added, may be within sampling noise. The authors should report uncertainties or at least explicitly discuss the statistical power limits before drawing conclusions about comparative performance.","section":"Section 3.3 and Tables 3-7"}],"minor_comments":[{"comment":"Grammar: 'evaluate these tools performance' should be 'evaluate these tools' performance'; 'addressed the following research questions' mixes tenses. The abstract would also benefit from stating the number of tools evaluated and the dataset size explicitly.","section":"Abstract and Section 1"},{"comment":"Typo: 'Similarlym, Greshake et al.' should be 'Similarly, Greshake et al.'","section":"Section 2.5"},{"comment":"The benchmark dataset is not made available: no link, repository, or data availability statement is provided. For a paper whose main contribution is a 'benchmark dataset,' releasing it (or an anonymized subset) is essential for reproducibility and for verifying the authors' own concern about training-data overlap.","section":"Section 3.2"},{"comment":"The column header 'PAI Guard' should be 'ProtectAI LLM Guard' for consistency with Tables 3-5. Also, 'LK Similarity' and 'LK Canary' should be expanded or defined in the caption, since the abbreviations are not introduced in the table.","section":"Tables 6 and 7"},{"comment":"Reference [79] in Section 3.2 is the same paper as [43] (Liu et al., 'Prompt Injection attack against LLM-integrated Applications'); duplicate citations should be merged.","section":"Reference [79]"},{"comment":"The table contains formatting issues, including a cut-off '/times✓' in the Output Scan. row and the header 'PAI Guard' (should be 'ProtectAI LLM Guard'). The legend says '-' for 'no information available,' but the cell for 'Jailbreak' in the ArthurAI column shows '-' which is inconsistent with the claim in the text that 'Arthur AI Firewall' provides jailbreak protection.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"This is a useful empirical study of an under-evaluated market, and the authors' willingness to state limitations is commendable. However, for a paper whose contribution is a benchmark, the lack of a released dataset and the absence of an explicit evaluation protocol for the 'best overall' claim are significant. The authors should be encouraged to either release the dataset (or a substantial curated subset) or reframe the paper as a limited-scope comparative evaluation rather than a benchmark paper. The ranking instability across prompt sources is the main technical risk, and the revision should address it head-on, not merely in a limitations paragraph."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. This is the first formal comparative evaluation of bolt-on LLM security tools, and it's worth reading for the market map alone. But the specific ranking—Lakera and ProtectAI as 'best overall'—should be treated as provisional, because the benchmark dataset isn't public and the paper itself shows the ranking shifts a lot depending on which prompt generator you use.\n\nWhat's genuinely good: the authors found 13 tools, got 7 evaluated, and used a plausible mix of manual and automated attack generation. They measured precision, recall, FPR, latency, and ASR across three conditions (no context, with context, and the Deepset set), and they document a real market transparency problem—four closed-source vendors wouldn't give access. The baseline finding is the most robust result: ChatGPT-3.5-Turbo has FPR between 0.438 and 0.859 depending on condition, which makes it practically unusable as a filter. That conclusion holds up across all their conditions and is not sensitive to the dataset composition issues.\n\nThe soft spots are real but mostly disclosed. Section 5.3 admits the attacks come from research tools and manual templates, not threat intelligence, so representativeness for real-world attacks is unknown. Tables 6 and 7 show exactly why this matters: ASR varies sharply by prompt source. For example, with base prompts, Lakera's ASR is 0% on Garak prompts but 83% on PromptMap prompts, and the baseline is 100% on manual prompts and 0% on Garak and PromptMap. So the winning set is not invariant across prompt sources; 'best overall' is an artifact of the particular 75.8%-malicious mix unless representativeness is established. On top of that, the benchmark and code are not released, there are no confidence intervals, and the 'best overall' claim is not tied to any defined composite metric. The convenience sample—only four proprietary tools that happened to provide trials—further limits the generality.\n\nThat said, this is a legitimate empirical study, honestly reported. It's not a formal benchmark in the style of Garak or InjecAgent; it's a product evaluation. The right use is as a starting point, not as a verdict. I'd want the dataset and scoring criteria released, along with a sensitivity analysis across prompt sources, before treating the rankings as durable. But the authors clearly engaged with the literature, disclosed their limitations, and provided a useful snapshot of an emerging market.\n\nWho it's for: practitioners choosing an LLM security tool, and researchers who want a baseline for future evaluations. It deserves a serious referee. I'd accept it with the expectation that the authors release the dataset and either define and justify a composite metric or temper the 'best overall' language.","headline":"Useful first cut at benchmarking commercial LLM security tools, but the headline rankings are not load-bearing on their own: the benchmark isn't released and the results are prompt-source-dependent.","tokens_in":18071,"tokens_out":1980,"would_cite":true,"duration_ms":21033,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper builds a 500-prompt benchmark of malicious and benign prompts, evaluates seven LLM safety filters and a ChatGPT-3.5-Turbo baseline, and reports that the baseline's false-positive rate (0.438–0.859) makes it unusable as a safety…","keywords":["LLM security","prompt injection","jailbreak","benchmark dataset","false positive rate","safety tool evaluation","LLM-integrated applications","prompt classification"],"falsifier":"Collect a corpus of malicious prompts actually observed against deployed LLM-integrated applications, for example from production logs or bug-bounty reports, run the same seven tools plus the baseline on it, and check whether Lakera Guard still has the lowest false-positive rate and highest F1 score and whether the baseline still exceeds a 0.4 false-positive rate; a substantial reordering would show the benchmark's results are artifacts of its synthetic prompt mix.","tokens_in":17049,"feed_emoji":"🛡️","tokens_out":8660,"duration_ms":84370,"temperature":0.7,"pith_summary":"This paper tries to establish whether bolt-on LLM security tools—filters that screen prompts before they reach a model—actually work and whether they are usable in production. To answer this, the authors assemble the Palit benchmark: 500 prompts spanning four LLM-integrated application contexts and four sources of attack generation, and evaluate seven commercial or open-source filters plus a ChatGPT-3.5-Turbo baseline. The central finding is that the raw baseline model is unsuitable as a safety filter because its false-positive rate ranges from 0.438 to 0.859, while the two best tools, Lakera Guard and ProtectAI LLM Guard, achieve high precision with far fewer false positives. A sympathetic reader would take away that LLM safety is not yet solved by any single tool and that context matters: detection changes when system prompts are added, and performance differs sharply across attack-generation sources.","feed_headline":"GPT-3.5 flags up to 86% of benign prompts as attacks","feed_subtitle":"A 500-prompt benchmark finds Lakera Guard and ProtectAI LLM Guard are the best of seven LLM safety filters.","key_machinery":"The load-bearing object is the benchmark itself: a collection of 500 prompts built from hand-written attacks and the outputs of three automated attack generators, paired with a benign-prompt set and an existing public dataset for cross-checking. Prompts are grounded in four application scenarios—French translation, a finance chatbot, a code-to-SQL tool, and a general-purpose chat agent—and are tested with and without the system prompt to isolate context sensitivity. The argument runs through standard detection metrics, including precision, recall, accuracy, false-positive rate, F1 score, latency, and attack success rate, so the tool ranking is an empirical comparison rather than a feature checklist.","core_discovery":"On its own terms, the paper claims that the emerging LLM safety-tool market has no obvious winner and that simple comparisons are misleading. The Palit benchmark evaluates seven filters—four open-source and three closed-source—plus ChatGPT-3.5-Turbo as a baseline. Lakera Guard posts the strongest combination across tests, with F1 scores of 0.659 without context, 0.809 with context, and 0.823 on an existing public prompt-injection dataset, along with sub-0.1-second latency; ProtectAI LLM Guard is close in precision but an order of magnitude slower. The baseline model often appears accurate because the dataset is imbalanced, yet its false-positive rates of 0.438–0.859 make it impractical. The paper also claims that the source of prompts matters: manually crafted attacks are the hardest to stop, while the behavior of generator-produced attacks changes sharply when context is added.","pith_inferences":["A testable extension would be to rerun the same seven tools on a corpus of attacks observed in real deployments, for example from production logs or bug-bounty reports; if the ranking changes, the synthetic prompt mix is not representative of field conditions.","The across-the-board improvement on the public prompt-injection dataset raises the possibility that several commercial filters have been trained or tuned on public benchmark data, which would make public-dataset results an upper bound rather than a field estimate.","The wide latency spread, from about 0.035 to over 29 seconds per prompt, implies an operational cost the paper leaves implicit: at production query volumes, latency alone may decide tool adoption even when a slower tool has marginally better detection.","The context ablation suggests that filter deployment should be tuned against the target application's actual prompt template, not validated in isolation."],"forward_implications":["Relying on a foundation model's built-in safety is not enough: at false-positive rates between 0.438 and 0.859, a production chatbot would flag or block a large share of legitimate requests.","If a deployer values both detection and low disruption, the results favor Lakera Guard and ProtectAI LLM Guard over the other five tested tools.","Context is a first-order variable: adding system prompts changed attack success rates sharply, including pushing manual prompts to 100% success against the baseline, so evaluations need application context to be meaningful.","Open-source tools in the study were hampered by outdated code and long latencies—one tool timed out on longer prompts and took seconds to minutes per prompt—so maintenance and speed are part of the safety story.","Accuracy is misleading on imbalanced attack/benign splits; false-positive rate and attack success rate should be reported alongside accuracy to judge a filter."],"supporting_citations":[{"why":"Supplies the existing benign and malicious prompt collection used as a second validation dataset for all tools.","marker":"[18]"},{"why":"Provides an automated security-probe framework whose attack templates form one of the prompt-generation sources in the benchmark.","marker":"[19]"},{"why":"Provides a context-specific black-box prompt-injection generation method used to create benchmark attacks.","marker":"[43]"},{"why":"Provides an automated attack-generation tool contributing direct, math, context-switching, and external-browsing prompts to the benchmark.","marker":"[62]"},{"why":"Provides the sandwich-defense prompt construction used for the ChatGPT-3.5-Turbo baseline classifier.","marker":"[7]"},{"why":"Supplies the standardized attack taxonomy that structures the tool-feature matrix and the threat model for prompt selection.","marker":"[69]"}],"fun_headline_variants":["Lakera Guard and ProtectAI LLM Guard lead LLM safety filter tests","No clear winner in seven LLM safety filters, new benchmark shows","Manual attacks are the hardest for LLM safety filters to catch","Benchmark reveals tradeoff: Lakera fast, ProtectAI precise but slow"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's malicious prompts are generated by researchers using manual techniques and automated attack tools, and the paper concedes it is unclear whether these resemble the attacks real adversaries would actually send to production LLM applications; if real-world attack distribution differs, the rankings may not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Lakera Guard and ProtectAI LLM Guard lead LLM safety filter tests","No clear winner in seven LLM safety filters, new benchmark shows","Manual attacks are the hardest for LLM safety filters to catch","Benchmark reveals tradeoff: Lakera fast, ProtectAI precise but slow"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00051,"raw_usage":{"total_tokens":2504,"prompt_tokens":989,"completion_tokens":1515,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":1436}},"tokens_in":605,"tokens_out":1515,"duration_ms":10592,"temperature":1.0,"reasoning_tokens":1436,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:21:10.059343+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a corpus of malicious prompts actually observed against deployed LLM-integrated applications, for example from production logs or bug-bounty reports, run the same seven tools plus the baseline on it, and check whether Lakera Guard still has the lowest false-positive rate and highest F1 score and whether the baseline still exceeds a 0.4 false-positive rate; a substantial reordering would show the benchmark's results are artifacts of its synthetic prompt mix.","supporting_citations":[{"cited_title":"deepset/prompt-injections · Datasets at Hugging Face — huggingface.co","cited_arxiv_id":null,"evidence_quote":"Supplies the existing benign and malicious prompt collection used as a second validation dataset for all tools."},{"cited_title":"GitHub - utkusen/promptmap: automatically tests prompt injection attacks on ChatGPT instances — github.com","cited_arxiv_id":null,"evidence_quote":"Provides an automated attack-generation tool contributing direct, math, context-switching, and external-browsing prompts to the benchmark."},{"cited_title":"https://learnprompting.org/docs/prompt_hacking/defensive_measures/ sandwich_defense, 2023","cited_arxiv_id":null,"evidence_quote":"Provides the sandwich-defense prompt construction used for the ChatGPT-3.5-Turbo baseline classifier."},{"cited_title":"Adversarial machine learning : a taxonomy and terminology of attacks and mitigations","cited_arxiv_id":null,"evidence_quote":"Supplies the standardized attack taxonomy that structures the tool-feature matrix and the threat model for prompt selection."}],"review_version":1}