Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

GenBreak: Red Teaming Text-to-Image Generators Using Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A red-team LLM bypasses image safety filters up to 89.7% of the time.

desk verdict GenBreak is a useful T2I red-teaming pipeline with notable commercial transfer results, but the headline rates depend on an unvalidated toxicity scorer that also serves as the training reward, so the absolute numbers should be read as upper bounds until the evaluator is independently validated. read the letter →

arxiv 2506.10047 v1 pith:5CYDJJDU submitted 2025-06-11 cs.CR cs.CL

Zilong Wang , Xiang Zheng , Xiaosen Wang , Bo Wang , Xingjun Ma , Yu-Gang Jiang This is my paper
classification cs.CRcs.CL
keywords text-to-imagesafetyredteamingjailbreakpromptsreinforcementlearningGRPOtoxicitybypassadversarialtransferattacks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes GenBreak, a framework that turns a small instruction-tuned language model into an automated red-teaming agent for text-to-image generators. It claims that, by combining supervised fine-tuning on curated attack prompts with reinforcement learning against a surrogate model, the resulting agent discovers adversarial prompts that are both stealthy and genuinely harmful. On safeguarded Stable Diffusion 2.1, GenBreak reports toxic bypass rates of 60.8% for nudity, 89.7% for violence, and 84.6% for hate, far above existing baselines. It also shows that these prompts transfer to commercial APIs in single attempts, with nudity bypass rates of 70%, 30%, and 47% against Leonardo.ai, fal.ai, and stability.ai. If this is right, attackers can automate discovery of high-risk prompts that current text and image filters miss, and defenders can use the same tool to find and patch vulnerabilities.

What carries the argument

The load-bearing object is a composite reward function used to train the red-team LLM by GRPO. The toxicity term is the average of three expert detectors—MHSC, LlavaGuard, and NudeNet—and this same average defines all reported toxicity and bypass metrics. A bypass reward is granted only when both the rewritten prompt and the resulting image pass every component of the integrated filter, and it is scaled by toxicity so that merely evading filters without producing harm earns no credit. The clean reward penalizes reliance on blacklisted sensitive words, while three diversity rewards based on SelfBLEU, sentence embeddings, and DreamSim keep the discovered prompts and images from collapsing into a few templates. This reward shaping is what lets the LLM learn innocuous-sounding paraphrases, such as indirect references that still trigger harmful outputs.

What would settle it

Have independent human annotators rate a random sample of GenBreak-generated images that scored above the 0.5 toxicity threshold, and compare their harm judgments to the detector scores; if agreement is far lower than the reported TBR and TCBR rates, the claim that GenBreak discovers genuinely high-risk prompts fails. A second check is to recompute toxicity after removing or replacing one of the three expert models: if scores drop below threshold for most images, the reported success depends on the specific evaluator rather than on intrinsic image harm.

Watch

Extended reading notes

Core claim

The central claim is that red-teaming text-to-image models can be fully automated by optimizing an LLM to generate prompts that simultaneously pass safety filters and yield images scored as highly toxic. GenBreak supervises the LLM on rewritten prompts from a category dataset and on high-scoring attack prompts collected from an uncensored generator, then applies GRPO with a composite reward: image toxicity from an aggregate of MHSC, LlavaGuard, and NudeNet; a bypass reward that pays only when both prompt and image pass all filter components; a clean reward for avoiding blacklisted words; and lexical, semantic, and image diversity rewards. On Stable Diffusion 2.1 with an integrated text-and-image filter, the method reaches toxic bypass rates of 60.8% (nudity), 89.7% (violence), and 84.6% (hate), and on Stable Diffusion 3 Medium it reports 80.7%, 89.5%, and 95.0%. Single-attempt transfer attacks on commercial services reached toxic bypass rates of 70% on Leonardo.ai, 30% on fal.ai, and 47% on stability.ai in the nudity domain, which the paper takes as evidence that prompts learned against a surrogate generalize to unknown production filters.

Load-bearing premise

The aggregated toxicity score from MHSC, LlavaGuard, and NudeNet is a valid and unbiased measure of how harmful an image actually is, because that score is both the RL reward and the definition of every reported bypass rate.

Editorial extensions

If this is right

  • Text- and keyword-based content filters can be evaded automatically; a small trained LLM generates bypassing prompts at scale, so static blacklists are not a sufficient defense.
  • Prompts trained against one open-source model transfer to undisclosed commercial filters, meaning a surrogate-based attack can probe production services without internal access.
  • Because GenBreak also uses a clean reward, its prompts evade keyword filters, so clean-bypass metrics should be part of standard safety evaluations of T2I systems.
  • The discovered prompts can serve as training data to harden filters, aligning red-teaming output with defensive fine-tuning of moderation systems.
  • The method requires image-toxicity scores even when the filter is triggered, which is available for open-source models and service providers but not for fully black-box attackers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: A natural next step the paper does not pursue is using the same reward design to mine prompts for other policy categories, such as deception, self-harm instructions, or copyrighted characters, by swapping the expert detectors and blacklist.
  • Inference: Because every headline number is defined by the aggregated detector scores, an independent human-panel validation of a sample of generated images would tell whether GenBreak discovered genuine harm or simply optimized the detectors; this is the most direct testable extension.
  • Inference: The strong transfer rates suggest commercial filters may share common failure modes with open-source safety checkers; if so, a GenBreak-style generator could be used by defenders to create a shared adversarial prompt benchmark across providers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes GenBreak, a two-stage framework that fine-tunes a red-team LLM to generate adversarial prompts for text-to-image (T2I) models. The first stage is supervised fine-tuning on curated rewrite and pre-attack datasets; the second stage uses GRPO reinforcement learning with a multi-component reward that combines image toxicity, filter bypass, avoidance of sensitive keywords, and lexical, semantic, and image diversity. The authors evaluate GenBreak on safeguarded Stable Diffusion 2.1 and 3 Medium with an integrated textual and visual filter, reporting toxic bypass rates of 60.8% (nudity), 89.7% (violence), and 84.6% (hate) on SD 2.1, and also report single-attempt transfer attacks against leonardo.ai, fal.ai, and stability.ai, with nudity-domain TBRs of 70%, 30%, and 47%, respectively. The paper concludes that GenBreak automatically discovers prompts that both evade safety filters and produce highly toxic images, exposing vulnerabilities in deployed T2I services.

Significance. If the reported results hold, GenBreak would be a valuable automated red-teaming tool for T2I systems, and the commercial transfer results would demonstrate practically concerning weaknesses in deployed safety mechanisms. The paper's strengths include a clearly described training pipeline, a reasonably extensive evaluation across multiple open-source models and three commercial APIs, and an ablation study of the reward components. The framework is a useful empirical contribution to the automated red-teaming literature. However, the central claim that the generated prompts produce 'genuinely toxic' images rests on a composite toxicity evaluator that is also the RL reward; the paper provides no independent validation of this evaluator, no held-out evaluation of generalization to new seed prompts, and no statistical confidence intervals. These gaps currently limit the strength of the headline claims.

major comments (4)
  1. [Section 3.3.2, Eq. (3), Section 4.1] The image toxicity evaluator Rtox is simultaneously the RL reward in Eq. (3) and the basis for every reported TBR/TCBR in Section 4.1. Because RL is optimized against this exact composite of MHSC, LlavaGuard, and NudeNet scores, the reported success rates partly measure the training objective rather than an independent notion of harm. The paper provides no human validation, no per-detector agreement statistics, and no false-positive analysis. Since the same evaluator scores the images returned by commercial APIs, the transfer rates inherit the same circularity. I request either a human rating study on a sample of generated images, a per-detector breakdown showing agreement, or evaluation with an external toxicity model that was not used in training.
  2. [Appendix C, Table 4, Section 4.1] For violence and hate, Rtox is the mean of only two detectors, MHSC and LlavaGuard, and the TBR/TCBR threshold is set to 0.5. With only two components, a single permissive detector can push the average above 0.5 even when the other detector strongly disagrees. For example, a LlavaGuard score of 1 and an MHSC score of 0.03 gives Rtox = 0.515, exceeding the threshold. The paper should report the per-detector toxic rates and their agreement (e.g., Cohen's kappa) for each category, and should show how TBR/TCBR vary when the threshold is varied independently for each detector.
  3. [Section 3.2, Section H.2] The evaluation protocol does not test generalization to unseen prompts. The seed set Dseed is used to build the Category Rewrite Dataset and the Pre-Attack Dataset (Section 3.2), to condition the RL policy (Section 3.3.1), and to generate the 5,000 evaluation prompts per category (Appendix H.2). The prompts transferred to commercial APIs are randomly sampled from the same evaluation pool (Appendix H.3). Thus, the open-source and transfer results measure performance on seed prompts already seen during training. To support the claim that GenBreak automatically discovers new adversarial prompts, the authors should hold out a portion of Dseed from both SFT and RL and evaluate on those held-out seeds.
  4. [Tables 1, 2, 7] No confidence intervals, standard errors, or significance tests are reported for any TBR, TCBR, or toxicity score. For the commercial transfer experiments, each cell is based on 100 prompts, so a reported TBR of 70% has a 95% confidence interval of roughly 60-79%; differences between methods of less than about 10 percentage points are not statistically distinguishable. The paper should report confidence intervals (e.g., Wilson intervals) for the headline rates and, where pairwise comparisons are made, a significance test.
minor comments (5)
  1. [Appendix C] The spellings 'LLaV AGuard' and 'LlavaGuard' are used inconsistently; the model should be referred to uniformly as 'LLaVA Guard' or 'LlavaGuard'.
  2. [Figures 2 and 4] The axis labels in Figures 2 and 4 contain the typo 'T oxicity' (extra space); these should read 'Toxicity Threshold'.
  3. [Figures 5-7 captions] The captions state 'We applied blurring and masked sensitive content using for ethical considerations.' The phrase 'using for' is incomplete and should be reworded, for example, '...using blurring and masking for ethical considerations.'
  4. [Section 2, Section 4.2] The related work section cites DACA, Atlas, and PromptTune as methods that improve attack success, but these methods are not included in the experimental comparison. The paper should either add these baselines or explicitly state why they are excluded.
  5. [Eq. (8), Appendix G.2/G.3] The main-text objective in Eq. (3) omits the gibberish penalty and symbol regulation reward; these appear only in Eq. (8) in the appendix. The main text should note that the full objective includes two additional fluency rewards, with details in the appendix.

Circularity Check

3 steps flagged · score 6.0 of 10

Open-source TBR/TCBR results are the RL training objective re-scored with the same toxicity evaluator, the same integrated filter, and the same Dseed, so Table 1 is largely self-evaluation; the commercial black-box transfer numbers remain a partly independent external check.

  1. fitted input called prediction [Section 3.3.2 (Eq. 3) vs. Section 4.1 (TBR) and Table 1]
    "Toxic Bypass Rate (TBR): This metric quantifies the percentage of adversarial prompts that successfully bypass the safety filters or mechanisms and generate images with toxicity scores above a predefined threshold τt. Formally, TBR = 1/N Σ I(filter(s_i)=“pass" ∧ R_tox(y_i) > τ_t), where I(·) is the indicator function."

    TBR is the headline result of Table 1. Its two conditions are exactly the two terms the RL objective maximizes: R_tox(y) is the λ1 toxicity reward in Eq. (3), and R_bypass(s,y)=R_tox(y)·I[bypass] is the λ2 term, with the same integrated filter used during training on the same surrogate SD 2.1. The evaluation also rewrites the same Dseed used for RL (Sec. H.2). Hence the reported TBR values (60.8%, 89.7%, 84.6%) are realized training rewards, not out-of-sample predictions.

  2. fitted input called prediction [Section 3.3.2 (Clean Reward) vs. Section 4.1 (TCBR) and Table 1]
    "Rclean(s) = Rtox(s)·I[f_blacklist(s) = 0], where f_blacklist detects explicit terms (blacklist in Appendix B) ... TCBR = 1/N Σ I(filter(s_i)=“pass" ∧ R_tox(y_i) > τ_t ∧ s_i ∩ B = ∅)."

    TCBR adds the condition s_i ∩ B = ∅, which is precisely the condition defining the clean reward R_clean(s) in the RL objective. The high TCBR values in Table 1 are therefore a count of how often the policy satisfies its own clean-reward constraint. This is a definitional reduction of the metric to a component of the training objective rather than an independently measured property.

1 more flagged steps
  1. fitted input called prediction [Section 3.2 (Pre-Attack Dataset) and Section H.2 (Evaluation Details)]
    "After completing T iterations for all seeds, we retain the top 20% of instances with the highest TBS in each risk category ... we generate 10 adversarial prompts for each seed example in Dseed, resulting in 5000 test cases per harmful category for evaluation."

    The Pre-Attack Dataset used for SFT is selected by TBS = I[bypass]·toxicity(y), with toxicity from the same R_tox evaluator (App. F), and the final evaluation generates prompts from the same Dseed. No held-out seed split or independent toxicity label is used, so the open-source TBR/TCBR numbers measure in-distribution reward maximization rather than generalization to new harmful concepts.

full rationale

The main circularity is the identity between the RL reward and the reported evaluation metrics. R_tox is the toxicity reward in Eq. (3), R_bypass is R_tox gated by the same integrated filter, and R_clean is R_tox gated by the same blacklist; TBR and TCBR are these same expressions with a threshold. Evaluating on Dseed, the same seed set used for training, removes even seed-level generalization from the open-source claims. The commercial transfer attacks (70%, 30%, 47% on nudity) are not circular in their bypass component, because the target filters are unknown and were never used in training, though the toxicity labels of returned images still use the same self-defined R_tox. There is no load-bearing self-citation chain or imported uniqueness theorem; citations to CRT and other prior work are independent. The paper's own Limitations statement concedes that the integrated filter may not reflect real commercial policies and that toxicity scores are needed as reward signals. Overall, partial circularity: score 6.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central results rest on a hand-tuned multi-objective reward, a specific toxicity threshold, and the assumed validity of the toxicity evaluator and the training-time filter; no new entities are introduced.

free parameters (5)
  • Reward weights lambda_1..lambda_8 = 1.0, 0.6, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0 (clean weight lambda_3 = 5 for SD3M nudity)
    Hand-chosen in Section G.4; they define the trade-off between toxicity, bypass, cleanliness, and diversity that the RL policy optimizes, so the central results depend on them.
  • Toxicity threshold tau_t = 0.5
    Used to define TBR and TCBR in Section 4.1; results change with the threshold as shown in Figure 2, so the headline rates are tied to this choice.
  • Dynamic reference pool size = 1000
    Controls how much history is used for lexical, semantic, and image diversity rewards; listed in Table 6.
  • Pre-Attack top-20% selection = 20%
    Retains only the highest-TBS attack attempts for SFT (Section 3.2, Appendix F); this filters the training data for the red-team model.
  • Number of evaluation prompts per seed = 10 (5000 per category)
    Evaluation protocol in Appendix H.2; the test set size affects stability of reported percentages.
assumptions (5)
  • domain assumption Aggregate toxicity scores from MHSC, LlavaGuard, and NudeNet accurately capture whether an image is harmful.
    Used as Rtox in Eq. (3) and as the metric in Section 4.1; no human validation is provided.
  • domain assumption The integrated filter (ParaDetox text classifier, NSFW text detector, and Stable Diffusion safety checker) is a representative proxy for real-world commercial content moderation.
    Section 4.1; the authors acknowledge in Limitations that it may not reflect actual commercial policies.
  • standard math GRPO and the RL objective are valid for optimizing the red-team policy.
    Section 3.3.1; standard policy-gradient theory is assumed.
  • domain assumption Adversarial prompts transfer across T2I models, so evaluation against commercial APIs measures real-world vulnerability.
    Section 4.3; no mechanistic explanation is provided beyond empirical transfer.
  • domain assumption The blacklist in Appendix B captures words that trigger safety filters in the tested systems.
    Used for both the clean reward and the TCBR metric; includes broad everyday words such as 'attack' and 'death', so its validity is not established.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GenBreak: Red Teaming Text-to-Image Generators Using Large Language Models." pith.science (2026). https://pith.science/paper/5CYDJJDU

@misc{pith2026250610047,
  author       = {Pith},
  title        = {Pith review of: GenBreak: Red Teaming Text-to-Image Generators Using Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5CYDJJDU}},
  note         = {Machine review of arXiv:2506.10047}
}
read the original abstract

Text-to-image (T2I) models such as Stable Diffusion have advanced rapidly and are now widely used in content creation. However, these models can be misused to generate harmful content, including nudity or violence, posing significant safety risks. While most platforms employ content moderation systems, underlying vulnerabilities can still be exploited by determined adversaries. Recent research on red-teaming and adversarial attacks against T2I models has notable limitations: some studies successfully generate highly toxic images but use adversarial prompts that are easily detected and blocked by safety filters, while others focus on bypassing safety mechanisms but fail to produce genuinely harmful outputs, neglecting the discovery of truly high-risk prompts. Consequently, there remains a lack of reliable tools for evaluating the safety of defended T2I models. To address this gap, we propose GenBreak, a framework that fine-tunes a red-team large language model (LLM) to systematically explore underlying vulnerabilities in T2I generators. Our approach combines supervised fine-tuning on curated datasets with reinforcement learning via interaction with a surrogate T2I model. By integrating multiple reward signals, we guide the LLM to craft adversarial prompts that enhance both evasion capability and image toxicity, while maintaining semantic coherence and diversity. These prompts demonstrate strong effectiveness in black-box attacks against commercial T2I generators, revealing practical and concerning safety weaknesses.

Figures

Figures reproduced from arXiv: 2506.10047 by the authors.

Figure 1
Figure 1. Overview of the proposed GenBreak framework. mechanisms, the resulting adversarial prompts often exhibit inconsistent effectiveness and typically require multiple attempts to generate highly harmful images. 3 Methodology Threat Model. We consider two threat models: (1) gray-box and (2) black-box. In the gray-box setting, which applies to open-source T2I models, the attacker can access the generated image regardless … view at source ↗
Figure 2
Figure 2. Performance of different algorithms (GenBreak, CRT, Vanilla RL) across toxicity thresholds [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Effects of different reward terms. Limitations Our work also has several limitations. First, training the red team LLM requires image toxicity scores as reward signals, which assumes access to this information regardless of whether the images bypass the filter. While this is feasible for open-source models or for T2I service providers, it remains challenging for black-box T2I models/services. Second, the integrated … view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Performance of different algorithms (GenBreak, CRT, Vanilla RL) across toxicity thresholds [PITH_FULL_IMAGE:figures/full_fig_p025_4.png]
Figure 5
Figure 5. Figure 5: Visualization of unsafe images generated by Leonardo.Ai and their corresponding attack [PITH_FULL_IMAGE:figures/full_fig_p026_5.png]
Figure 6
Figure 6. Figure 6: Visualization of unsafe images generated by fal.ai and their corresponding attack prompts [PITH_FULL_IMAGE:figures/full_fig_p027_6.png]
Figure 7
Figure 7. Figure 7: Visualization of unsafe images generated by stability.ai and their corresponding attack [PITH_FULL_IMAGE:figures/full_fig_p027_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Trojan Horse Prompting: Jailbreaking Conversational Multimodal Models by Forging Assistant Message

    cs.AI 2025-07 reject novelty 5.0 of 10

    Trojan Horse Prompting injects malicious instructions into a fabricated assistant message in the API chat history, aiming to bypass Gemini's safety filters, but no quantitative evidence is provided.

Reference graph

Works this paper leans on

38 extracted references · 25 canonical work pages · cited by 1 Pith paper

  1. [1]

    URL https://huggingface.co/CompVis/ stable-diffusion-safety-checker

    Stable diffusion safety checker. URL https://huggingface.co/CompVis/ stable-diffusion-safety-checker

    work page

  2. [2]

    URLhttps://bfl.ai/announcements/24-08-01-bfl

    Flux.1. URLhttps://bfl.ai/announcements/24-08-01-bfl

    work page

  3. [3]

    URLhttps://fal.ai/

    Official website of fal.ai. URLhttps://fal.ai/

    work page

  4. [4]

    URL https://cloud.google.com/vertex-ai/generative-ai/docs/ models/gemini/2-0-flash

    Gemini 2.0 flash. URL https://cloud.google.com/vertex-ai/generative-ai/docs/ models/gemini/2-0-flash

    work page

  5. [5]

    URLhttps://leonardo.ai/

    Leonardo.ai official website. URLhttps://leonardo.ai/

    work page

  6. [6]

    URL https://huggingface.co/Orenguteng/ Llama-3.1-8B-Lexi-Uncensored-V2

    Llama-3.1-8b-lexi-uncensored-v2. URL https://huggingface.co/Orenguteng/ Llama-3.1-8B-Lexi-Uncensored-V2

    work page

  7. [7]

    URL https://huggingface.co/eliasalbouzidi/ distilbert-nsfw-text-classifier

    Nsfw text detector. URL https://huggingface.co/eliasalbouzidi/ distilbert-nsfw-text-classifier

    work page

  8. [8]

    URLhttps://github.com/notAI-tech/NudeNet

    Nudenet. URLhttps://github.com/notAI-tech/NudeNet

    work page

Show all 38 references
  1. [9]

    URL https://huggingface.co/stabilityai/ stable-diffusion-2-1

    Stable diffusion 2.1. URL https://huggingface.co/stabilityai/ stable-diffusion-2-1

  2. [10]

    URL https://huggingface.co/stabilityai/ stable-diffusion-3-medium

    Stable diffusion 3 medium. URL https://huggingface.co/stabilityai/ stable-diffusion-3-medium

  3. [11]

    URLhttps://stability.ai/

    Stability ai official website. URLhttps://stability.ai/

  4. [12]

    Divide-and-conquer attack: Harnessing the power of llm to bypass safety filters of text-to-image models.arXiv preprint arXiv:2312.07130, 2023

    Yimo Deng and Huangxun Chen. Divide-and-conquer attack: Harnessing the power of llm to bypass safety filters of text-to-image models.arXiv preprint arXiv:2312.07130, 2023

    2023 arXiv

  5. [13]

    Jailbreaking text-to- image models with llm-based agents.arXiv preprint arXiv:2408.00523, 2024

    Yingkai Dong, Zheng Li, Xiangtao Meng, Ning Yu, and Shanqing Guo. Jailbreaking text-to- image models with llm-based agents.arXiv preprint arXiv:2408.00523, 2024

    2024 arXiv

  6. [14]

    Dreamsim: Learning new dimensions of human visual similarity using synthetic data

    Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. InAdvances in Neural Information Processing Systems, volume 36, pages 50742–50768, 2023

    2023

  7. [15]

    Llava- guard: Vlm-based safeguard for vision dataset curation and safety assessment

    Lukas Helff, Felix Friedrich, Manuel Brack, Patrick Schramowski, and Kristian Kersting. Llava- guard: Vlm-based safeguard for vision dataset curation and safety assessment. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pa...

    2024

  8. [16]

    Glass, Akash Srivastava, and Pulkit Agrawal

    Zhang-Wei Hong, Idan Shenfeld, Tsun-Hsuan Wang, Yung-Sung Chuang, Aldo Pareja, James R. Glass, Akash Srivastava, and Pulkit Agrawal. Curiosity-driven red-teaming for large language models. InThe Twelfth International Conference on Learning Representations, 2024. URL https://op...

    2024

  9. [17]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview. net/forum?id=nZeVKeeFYf9

    2022

  10. [18]

    Perception-guided jailbreak against text-to-image models

    Yihao Huang, Le Liang, Tianlin Li, Xiaojun Jia, Run Wang, Weikai Miao, Geguang Pu, and Yang Liu. Perception-guided jailbreak against text-to-image models. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 26238–26247, 2025

    2025

  11. [19]

    Jailbreaking safeguarded text-to-image models via large language models.arXiv preprint arXiv:2503.01839, 2025

    Zhengyuan Jiang, Yuepeng Hu, Yuchen Yang, Yinzhi Cao, and Neil Zhenqiang Gong. Jailbreaking safeguarded text-to-image models via large language models.arXiv preprint arXiv:2503.01839, 2025

    2025

  12. [20]

    Art: Automatic red-teaming for text-to-image models to protect benign users.arXiv preprint arXiv:2405.19360, 2024

    Guanlin Li, Kangjie Chen, Shudong Zhang, Jie Zhang, and Tianwei Zhang. Art: Automatic red-teaming for text-to-image models to protect benign users.arXiv preprint arXiv:2405.19360, 2024

    2024 arXiv

  13. [21]

    Arondight: Red teaming large vision language models with auto-generated multi-modal jailbreak prompts

    Yi Liu, Chengjun Cai, Xiaoli Zhang, Xingliang Yuan, and Cong Wang. Arondight: Red teaming large vision language models with auto-generated multi-modal jailbreak prompts. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 3578–3586, 2024. 10

    2024

  14. [22]

    Paradetox: Detoxifi- cation with parallel data

    Varvara Logacheva, Daryna Dementieva, Sergey Ustyantsev, Daniil Moskovskiy, David Dale, Irina Vladimirovna Krotova, Nikita Semenov, and Alexander Panchenko. Paradetox: Detoxifi- cation with parallel data. InAnnual Meeting of the Association for Computational Linguistics, 2022

    2022

  15. [23]

    FLIRT: Feedback loop in-context red teaming

    Ninareh Mehrabi, Palash Goyal, Christophe Dupuy, Qian Hu, Shalini Ghosh, Richard Zemel, Kai-Wei Chang, Aram Galstyan, and Rahul Gupta. FLIRT: Feedback loop in-context red teaming. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7...

    2024

  16. [24]

    Llama 3.2: Revolutionizing edge AI and vision with open, customizable models, 2024

    Meta AI. Llama 3.2: Revolutionizing edge AI and vision with open, customizable models, 2024. URL https://ai.meta.com/blog/ llama-3-2-connect-2024-vision-edge-mobile-devices/

    2024

  17. [25]

    Red teaming language models with language models

    Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. InConference on Empirical Methods in Natural Language Processing, 2022

    2022

  18. [26]

    Unsafe diffusion: On the generation of unsafe images and hateful memes from text-to-image models

    Yiting Qu, Xinyue Shen, Xinlei He, Michael Backes, Savvas Zannettou, and Yang Zhang. Unsafe diffusion: On the generation of unsafe images and hateful memes from text-to-image models. InProceedings of the 2023 ACM SIGSAC conference on computer and communications security, pages...

    2023

  19. [27]

    Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    2019

  20. [28]

    Red-teaming the stable diffusion safety filter.arXiv preprint arXiv:2210.04610, 2022

    Javier Rando, Daniel Paleka, David Lindner, Lennart Heim, and Florian Tramèr. Red-teaming the stable diffusion safety filter.arXiv preprint arXiv:2210.04610, 2022

    2022 arXiv

  21. [29]

    Sentence-bert: Sentence embeddings using siamese bert- networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. InConference on Empirical Methods in Natural Language Processing, 2019. URL https://api.semanticscholar.org/CorpusID:201646309

    2019

  22. [30]

    High- resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  23. [31]

    Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models

    Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22522–22531, 2023

    2023

  24. [32]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  25. [33]

    Ring-a-bell! how reliable are concept removal methods for diffusion models? InThe Twelfth International Conference on Learning Representations,

    Yu-Lin Tsai, Chia-Yi Hsu, Chulin Xie, Chih-Hsun Lin, Jia You Chen, Bo Li, Pin-Yu Chen, Chia-Mu Yu, and Chun-Ying Huang. Ring-a-bell! how reliable are concept removal methods for diffusion models? InThe Twelfth International Conference on Learning Representations,

  26. [34]

    Mma- diffusion: Multimodal attack on diffusion models

    Yijun Yang, Ruiyuan Gao, Xiaosen Wang, Tsung-Yi Ho, Nan Xu, and Qiang Xu. Mma- diffusion: Multimodal attack on diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7737–7746, 2024

    2024

  27. [35]

    Sneakyprompt: Jailbreaking text-to-image generative models

    Yuchen Yang, Bo Hui, Haolin Yuan, Neil Gong, and Yinzhi Cao. Sneakyprompt: Jailbreaking text-to-image generative models. In2024 IEEE symposium on security and privacy (SP), pages 897–912. IEEE, 2024

    2024

  28. [36]

    Natural language induced adversarial images

    Xiaopei Zhu, Peiyang Xu, Guanning Zeng, Yinpeng Dong, and Xiaolin Hu. Natural language induced adversarial images. InProceedings of the 32nd ACM International Conference on Multimedia, pages 10872–10881, 2024

    2024

  29. [37]

    messages

    Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. InThe 41st international ACM SIGIR conference on research & development in information retrieval, pages 1097–1100, 2018. 11 Appendice...

    2018

  30. [2024]

    URLhttps://openreview.net/forum?id=lm7MRcsFiS

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.