GenAI Security: Outsmarting the Bots with a Proactive Testing Framework

Gopala Dhar; Lavi Nigam; Sunil Kumar Jang Bahadur

REVIEW 4 major objections 5 minor 6 references

GenAI Security: Outsmarting the Bots with a Proactive Testing Framework

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An automated red/blue agent loop catches prompt injections with 0.985 F1.

desk verdict A practical agentic red/blue teaming framework whose reported evaluation measures the wrong thing; the paper is honest about its proxy but then overclaims empirical proof. read the letter →

arxiv 2505.18172 v1 pith:6E7FRK2Z submitted 2025-05-14 cs.CR cs.LG

Sunil Kumar Jang Bahadur , Gopala Dhar , Lavi Nigam This is my paper · ORCID

classification cs.CRcs.LG

keywords GenAIsecuritypromptinjectionredteamingblueagenticframeworkLLMtestingproactiveSPMLdataset

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a fully automated, two-agent testing framework can move GenAI security from reactive patching to proactive testing. A Red Teaming agent probes chatbot system prompts for injection vulnerabilities, and a Blue Teaming agent recommends mitigation services for attacks that succeed. Using the SPML Chatbot Prompt Injection Dataset, the authors report that the Red Teaming agent identifies injected user prompts with an F1 of 0.985, and that the Blue Teaming agent's number of recommendations rises with the dataset's severity score. A sympathetic reader would take the central claim to be that a generative-model-driven red/blue team loop can be both accurate enough and scalable enough for real-world GenAI application security.

What carries the argument

The load-bearing mechanism is the two-agent architecture: a Red Teaming agent that uses a knowledge base of prompt-attack techniques (SurrogatePrompt, SneakyPrompt) and a shared short-/long-term memory to generate or recognize injection attempts, and a Blue Teaming agent that consumes the Red agent's findings and returns a set of mitigation recommendations. What carries the argument is the proxy evaluation loop: the Red agent's output is scored as a binary classifier by an external evaluator model, and the Blue agent is calibrated by comparing recommendation count against the dataset's Degree field. The framework's claims of automation and adaptability rest on this loop's ability to feed new attack knowledge back into the agents.

What would settle it

Take the same Red Teaming agent's flagged and unflagged prompts from the SPML dataset and have independent human security reviewers label them under the same system-prompt/user-prompt pairing; if human-judged true positive rate on the agent's 'no injection' calls is materially higher than the proxy evaluator's implied false-negative rate, the 0.985 F1 overstates the agent's real detection ability.

Watch

Extended reading notes

Core claim

The central discovery is that an agentic red-team/blue-team loop, with each agent built on a large language model and connected through shared memory and a knowledge base of known attack techniques, can detect prompt injections and suggest proportional mitigations without human involvement at test time. The authors reframe Red-agent evaluation as a binary classification problem: given a system prompt and a user prompt, the agent decides whether the user prompt is an injection, and an external evaluator model checks that decision against the dataset's labels. Reported results are an F1 of 0.985 (precision 0.9951, recall 0.9751) over the 16K-record SPML dataset, with more false negatives than false positives. For the Blue agent, the paper reports a positive association between the dataset's Degree severity field and the number of mitigation recommendations, and interprets both results as evidence that the framework is suitable for real-world proactive testing.

Load-bearing premise

The measured effectiveness rests on accepting the external evaluator model's binary labels and the dataset's Degree ratings as accurate stand-ins for whether a prompt injection would actually succeed and how much defense it needs.

Editorial extensions

If this is right

If the framework works as reported, organizations can run prompt-injection testing continuously instead of at snapshot points, keeping defenses current as new bypass techniques appear.
Because both agents are driven by the same generative model family, the testing loop can scale across many system prompts without requiring the same human expert hours that traditional red teams demand.
The measured F1 of 0.985 implies the Red Teaming agent can surface the large majority of injected prompts while keeping false alarms low enough for practical screening.
The observed severity-to-recommendation correlation gives the Blue Teaming agent a way to focus mitigation effort on the attacks that matter most, rather than treating all injections equally.
The paper's framework is designed to be extended: shared memory and the knowledge base allow newly discovered attacks to be folded back into future test runs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the paper's proxy evaluation measures classification agreement with the dataset's labels, not whether an attack would actually succeed against a deployed application; a field trial against live chatbots would be the direct test of the framework's real-world claim.
Editorial inference: the same two-agent loop could be applied to other GenAI failure modes, such as data poisoning, jailbreaks, or hallucination-inducing prompts, wherever a severity-graded labeled corpus exists, since neither the architecture nor the evaluation logic is injection-specific.
Editorial inference: the Blue Teaming agent's 'more recommendations for worse attacks' result suggests the recommendation list could be turned into a risk-scoring signal that automatically prioritizes which vulnerabilities to patch first; the paper does not make that prioritization claim.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should read this as an engineering write-up, not as a scientific validation. The core idea—two GenAI agents, one generating injection attempts, one recommending mitigations, with a shared memory and knowledge base—is sensible and roughly what practitioners are already building. The paper does several things well: it cites the critical literature (including Feffer et al. on red teaming as "security theater"), it uses a public dataset (SPML), and it is unusually candid in Section IV-B that the red team evaluation is a proxy because there is no good open benchmark for generated injections. That honesty deserves credit.

The soft spot is load-bearing. The proxy converts the red team agent's evaluation into a binary classification task: can the agent label an existing user prompt as an injection for a given system prompt. The reported F1 of 0.985 therefore measures detection, not generation. It says nothing about whether the agent can craft a prompt that makes a target model deviate, bypass safety filters, or achieve an attack objective. The paper's own example in IV-C confirms this: the failure case is the agent misclassifying a nuanced injection, not failing to generate one. The blue team result is even thinner—a density scatter plot with a hand-drawn "red dotted line" claiming positive correlation between an ordinal "Degree" field and the count of recommendations from a fixed list of four. No correlation coefficient, no confidence interval, no baseline. The abstract and conclusion say the framework is "empirically proven" and "robust, balanced, and well-suited for real-world applications." That is not what the experiments show.

I would not call this a fraudulent or incoherent paper. The authors know they used a proxy and say so. The weakness is that the conclusions do not respect that limitation. The citation pattern looks fine; self-citation is not an issue. No code or data is shipped, but the dataset is public and the metrics are reproducible enough.

Who gets value? A developer looking for a high-level architecture sketch for agentic security testing might find the system description useful. A researcher evaluating security claims will not learn much from the numbers. I would not cite it in my own work on LLM security evaluation, but I would send it to a workshop as a "systems and practice" submission with strong encouragement to either fix the evaluation or reframe the claims as a demo, not a proof. It deserves a serious referee only if the venue is explicitly practice-oriented; for a research venue, the current evidence does not support the central claim.

Referee Report

4 major / 5 minor

Summary. The paper proposes an agentic "proactive testing framework" for GenAI security that combines a Red Teaming agent (tasked with generating prompt injection attacks) and a Blue Teaming agent (tasked with recommending mitigations), supported by a shared memory and knowledge base. The authors evaluate the framework on the SPML Chatbot Prompt Injection Dataset. The Red Teaming agent is evaluated via an indirect proxy: an evaluator model (gemini-1.5) uses a binary classification task to check whether the agent can label a user prompt as an injection for a given system prompt, achieving an F1 score of 0.985. The Blue Teaming agent is evaluated by comparing the dataset's "Degree" severity field with the number of mitigation recommendations (chosen from a fixed list of 4), and the paper claims a strong positive correlation based on a density scatter plot.

Significance. If the framework actually worked as described, a unified, automated red/blue teaming pipeline for GenAI applications would be a useful contribution to security practice. However, the paper's central empirical claim—that the framework is "effective" and "well-suited for real-world applications"—rests entirely on an indirect proxy evaluation that does not measure the claimed capabilities. The paper provides no direct evidence that prompts generated by the Red Teaming agent cause target GenAI models to deviate from instructions or bypass safety filters, and no statistical quantification for the Blue Teaming correlation. The conceptual architecture is described only at a high level, with no implementation details or code, limiting reproducibility. The potential significance is therefore not realized in the current manuscript.

major comments (4)

[Section IV-B, Table I] The Red Teaming agent's objective, stated in Section III, is to generate prompt injection attacks that circumvent safety filters. The evaluation, however, measures the agent's ability to perform binary classification of dataset prompts into "injection" vs. "no injection" via an evaluator model. The F1 score of 0.985 therefore characterizes this classification proxy, not the agent's attack generation ability. The paper even acknowledges this by calling it a "proxy" (Section IV-B) and by giving an example in Section IV-C of the agent failing to classify a nuanced injection. No experiment tests whether prompts actually generated by the Red Teaming agent induce any target GenAI system to behave maliciously. This is load-bearing because the abstract and conclusion claim empirical proof of the framework's effectiveness.
[Section IV-B, Fig. 3] The Blue Teaming evaluation claims a "strong positive correlation" between the attack severity (Degree field) and the number of recommendations made by the agent, but no correlation coefficient, confidence interval, or statistical test is reported. The density scatter plot and hand-drawn trend line are not a quantitative substitute. Additionally, the number of recommendations from a fixed list of four is a weak proxy for mitigation quality; recommending more services does not necessarily mean the mitigation is more effective. The claim in Section V that the agent "showed a clear understanding of the severity of prompt injections" is therefore unsupported.
[Section IV-B and Section III] The evaluation methodology is circular with respect to the Red Teaming agent's role: the agent is asked to generate attacks, but the proxy evaluates it as a detector. The paper's own framing in Section IV-B transforms the problem into "binary classification," which is a different task than attack generation. This mismatch means that even a perfect F1 score would not justify the conclusion that the framework can "outsmart the bots" or "generate and investigate potential scenarios" as claimed in the architecture section. The evaluation does not measure any adversarial success metric, such as attack success rate against a target model.
[Section III and IV] The architecture lacks the detail needed to reproduce or verify the claimed results. The paper does not specify the prompt templates used for the agents, the contents or construction of the Knowledge Base, the implementation of shared memory, the exact evaluator prompt, or the number of trials. Without these details, the empirical results in Tables I and II and Figure 3 cannot be independently checked, which is a significant shortcoming for a paper whose main contribution is an empirically validated framework.

minor comments (5)

[Section IV-C, Table II] The confusion matrix shows 312 false negatives out of 12,533 ground-truth injection cases; the paper should discuss whether these are concentrated in particular attack types or severity levels, since this could affect the interpretation of the model's practical utility.
[Section II, list of limitations] The limitations listed for traditional red/blue teaming (stagnation, scalability, scope, resource intensity, reactivity, human bias) are asserted without citations; providing references or concrete examples would strengthen the motivation.
[References] The references are a mix of relevant prior work (e.g., SurrogatePrompt, SneakyPrompt, SPML, Red-Teaming for Generative AI), but the paper does not position itself relative to existing automated red teaming frameworks, such as those using reinforcement learning or adversarial prompting; adding a brief related-work discussion would help contextualize the contribution.
[General] There are several typographical and clarity issues, including "tøinvestigate" in Section III, the undefined acronym SPML in the abstract (defined only in Section IV-A), and Figure 1 appearing before its first mention in the text.
[Section IV-C, Fig. 3] The density scatter plot is referenced but not shown in a way that allows the reader to assess the claimed trend; the red dotted line should be derived from a stated regression or trend model rather than drawn informally.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the evaluation is anchored by external SPML ground-truth labels and Degree values; the proxy mismatch is a validity limitation, not a self-referential derivation.

full rationale

The paper's central empirical claims are evaluated against an external benchmark dataset (SPML) with pre-existing ground-truth labels and a Degree field. The Red Teaming F1 score is a measured binary-classification performance computed by comparing evaluator-model outputs to those labels; nothing in the metric is defined in terms of the conclusion, and no parameter is fitted from the target outcome. The proxy design in Section IV-B explicitly redefines the evaluation task as binary classification, which is a validity limitation rather than a circularity: the reported F1 is evidence about the proxy classifier's agreement with ground truth, not about attack-generation success. Similarly, the Blue Teaming comparison correlates the dataset's Degree field with the number of selected recommendations; the hypothesis is tested against data, not constructed from the outcome. The paper contains no load-bearing self-citations (the references are external works), no uniqueness theorem imported from the authors, and no ansatz smuggled in via citation. The conclusion overstates what the proxy supports, but that is an inference-strength problem, not a derivation that reduces to its own inputs.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central evaluation rests on several unverified domain assumptions, including the correctness of the dataset labels, the reliability of the gemini-1.5 evaluator, and the validity of the severity proxy. The free parameter max_recommendations caps the Blue Teaming output and shapes the correlation claim.

free parameters (1)

max_recommendations = 4
The Blue Teaming agent is only allowed to choose from a fixed list of 4 recommendations, capping the maximum output. This hand-chosen limit constrains the reported correlation and is not derived from data.

assumptions (4)

domain assumption The SPML Chatbot Prompt Injection Dataset provides accurate ground truth for prompt injection detection.
The entire Red Teaming evaluation uses the dataset's binary labels as ground truth without independent verification of label correctness.
domain assumption The gemini-1.5 evaluator model reliably determines whether a user prompt is an injection for a given system prompt.
The Red Teaming evaluation is reduced to a binary classification problem solved by an evaluator model; no accuracy of the evaluator itself is reported.
domain assumption The 'Degree' field in the dataset is a valid measure of prompt injection severity and correlates with mitigation effort.
The Blue Teaming evaluation assumes that a higher degree of injection should correspond to more recommendations; this assumption is not independently validated.
ad hoc to paper A classification-based proxy can measure the effectiveness of a Red Teaming agent whose objective is to generate attacks.
The paper explicitly states it uses a proxy because of limitations in evaluation methodology, but the proxy only tests the agent's ability to classify injections, not to generate novel or evasive attacks. This is an ad hoc assumption central to the claimed results.

how reviews work

0 comments

Cite this review

Pith. "Pith review of GenAI Security: Outsmarting the Bots with a Proactive Testing Framework." pith.science (2026). https://pith.science/paper/6E7FRK2Z

@misc{pith2026250518172,
  author       = {Pith},
  title        = {Pith review of: GenAI Security: Outsmarting the Bots with a Proactive Testing Framework},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6E7FRK2Z}},
  note         = {Machine review of arXiv:2505.18172}
}

read the original abstract

The increasing sophistication and integration of Generative AI (GenAI) models into diverse applications introduce new security challenges that traditional methods struggle to address. This research explores the critical need for proactive security measures to mitigate the risks associated with malicious exploitation of GenAI systems. We present a framework encompassing key approaches, tools, and strategies designed to outmaneuver even advanced adversarial attacks, emphasizing the importance of securing GenAI innovation against potential liabilities. We also empirically prove the effectiveness of the said framework by testing it against the SPML Chatbot Prompt Injection Dataset. This work highlights the shift from reactive to proactive security practices essential for the safe and responsible deployment of GenAI technologies

Figures

Figures reproduced from arXiv: 2505.18172 by the authors.

**Figure 1.** Proactive testing framework approach However, given the rapid release of new GenAI models and the constant discovery of new bypass techniques, traditional Red Teaming and Blue Teaming approaches face several limitations, some of those are listed below: • Stagnates Fast: Traditional approaches provide solutions at a snapshot in time; such a security posture can quickly become outdated. • Difficult to Scale: These sol… view at source ↗

**Figure 2.** GenAI testing framework architecture In future work, the collected data and scenarios can be later used to further fine-tune the all models for better understanding and performance. Given a particular use case and specific vulnerabilities that need to be tested, the ”Red Teaming” agent will generate and investigate potential scenarios as part of an end-to-end workflow. The agent will then share its findings with the… view at source ↗

**Figure 3.** Blue Teaming Agent Evaluation Plot TABLE II CONFUSION MATRIX FOR RED TEAMING AGENT EVALUATION Predicted No Injection Predicted Injection Ground Truth No Injection 3410 60 Ground Truth Injection 312 12221 Upon plotting a density scatter plot of the recommendations suggested by the Blue Teaming agent vs the degree of injection severity, in [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

6 extracted references · 1 canonical work pages

[1]

arXiv:2307.10490 [cs.CR], 3rd Oct 2023

Eugene Bagdasaryan, Tsung-Yin Hsieh, Ben Nassi, Vitaly Shmatikov, Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs. arXiv:2307.10490 [cs.CR], 3rd Oct 2023

arXiv 2023
[2]

Lipton, Hoda Heidari, Red-Teaming for Generative AI: Silver Bullet or Security Theater? arXiv:2401.15897 [cs.CY], 27th Aug 2024

Michael Feffer, Anusha Sinha, Wesley Hanwen Deng, Zachary C. Lipton, Hoda Heidari, Red-Teaming for Generative AI: Silver Bullet or Security Theater? arXiv:2401.15897 [cs.CY], 27th Aug 2024

arXiv 2024
[3]

arXiv:2309.14122 [cs.CV], 17th Oct 2024

Zhongjie Ba, Jieming Zhong, Jiachen Lei, Peng Cheng, Qinglong Wang, Zhan Qin, Zhibo Wang, Kui Ren, SurrogatePrompt: Bypassing the Safety Filter of Text-to-Image Models via Substitution. arXiv:2309.14122 [cs.CV], 17th Oct 2024

arXiv 2024
[4]

arXiv:2305.12082 [cs.LG], 10th Nov 2023

Yuchen Yang, Bo Hui, Haolin Yuan, Neil Gong, Yinzhi Cao, SneakyPrompt: Jailbreaking Text-to-image Generative Models. arXiv:2305.12082 [cs.LG], 10th Nov 2023

arXiv 2023
[5]

arXiv:2402.11755 [cs.LG], 19th Feb 2024

Reshabh K Sharma, Vinayak Gupta, Dan Grossman, SPML: A DSL for Defending Language Models Against Prompt Attacks. arXiv:2402.11755 [cs.LG], 19th Feb 2024

arXiv 2024
[6]

UA O ] e R]U 'A @RVV&wͅ9s @ +p >w^VbZ= ]rc #fep e4s; @ @ @ fG`o\蜝 r @ @ @+W, Ñpx @ @ @ R 3ϵm( N 6

11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthor...

work page

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

arXiv:2307.10490 [cs.CR], 3rd Oct 2023

Eugene Bagdasaryan, Tsung-Yin Hsieh, Ben Nassi, Vitaly Shmatikov, Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs. arXiv:2307.10490 [cs.CR], 3rd Oct 2023

arXiv 2023

[2] [2]

Lipton, Hoda Heidari, Red-Teaming for Generative AI: Silver Bullet or Security Theater? arXiv:2401.15897 [cs.CY], 27th Aug 2024

Michael Feffer, Anusha Sinha, Wesley Hanwen Deng, Zachary C. Lipton, Hoda Heidari, Red-Teaming for Generative AI: Silver Bullet or Security Theater? arXiv:2401.15897 [cs.CY], 27th Aug 2024

arXiv 2024

[3] [3]

arXiv:2309.14122 [cs.CV], 17th Oct 2024

Zhongjie Ba, Jieming Zhong, Jiachen Lei, Peng Cheng, Qinglong Wang, Zhan Qin, Zhibo Wang, Kui Ren, SurrogatePrompt: Bypassing the Safety Filter of Text-to-Image Models via Substitution. arXiv:2309.14122 [cs.CV], 17th Oct 2024

arXiv 2024

[4] [4]

arXiv:2305.12082 [cs.LG], 10th Nov 2023

Yuchen Yang, Bo Hui, Haolin Yuan, Neil Gong, Yinzhi Cao, SneakyPrompt: Jailbreaking Text-to-image Generative Models. arXiv:2305.12082 [cs.LG], 10th Nov 2023

arXiv 2023

[5] [5]

arXiv:2402.11755 [cs.LG], 19th Feb 2024

Reshabh K Sharma, Vinayak Gupta, Dan Grossman, SPML: A DSL for Defending Language Models Against Prompt Attacks. arXiv:2402.11755 [cs.LG], 19th Feb 2024

arXiv 2024

[6] [6]

UA O ] e R]U 'A @RVV&wͅ9s @ +p >w^VbZ= ]rc #fep e4s; @ @ @ fG`o\蜝 r @ @ @+W, Ñpx @ @ @ R 3ϵm( N 6

11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthor...

work page