pith. sign in

arxiv: 2407.21772 · v2 · pith:R3G4IDJ3new · submitted 2024-07-31 · 💻 cs.CL · cs.LG

ShieldGemma: Generative AI Content Moderation Based on Gemma

Wenjun Zeng , Yuchi Liu , Ryan Mullins , Ludovic Peran , Joe Fernandez , Hamza Harkous , Karthik Narasimhan , Drew Proud
show 4 more authors
Piyush Kumar Bhaktipriya Radharapu Olivia Sturman Oscar Wahltinez
This is my paper

Pith reviewed 2026-05-20 13:12 UTC · model grok-4.3

classification 💻 cs.CL cs.LG
keywords content moderationLLM safetysafety classificationGemmasynthetic dataharm detectiongenerative AI
0
0 comments X

The pith

ShieldGemma models built on Gemma2 deliver more accurate safety risk predictions than prior systems for both user inputs and generated outputs.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents ShieldGemma as a suite of models for identifying safety risks such as sexually explicit content, dangerous material, harassment, and hate speech. It seeks to establish that these models outperform existing ones like Llama Guard and WildCard on both public and internal tests while relying mainly on synthetic training data. If the results hold, developers could integrate more reliable filters into generative AI systems to reduce harmful outputs. The work also introduces an LLM-driven process for creating and labeling safety data that generalizes well. Releasing the models aims to support broader efforts in making AI outputs safer.

Core claim

ShieldGemma models achieve state-of-the-art predictions of safety risks across key harm types and demonstrate superior performance compared to existing models such as Llama Guard (+10.8% AU-PRC on public benchmarks) and WildCard (+4.3%). The models handle both user input and LLM-generated output, with strong results even when trained primarily on synthetic data produced by a new curation pipeline.

What carries the argument

The ShieldGemma suite of models built upon Gemma2, paired with an LLM-based data curation pipeline that generates and labels training examples for safety classification tasks.

If this is right

  • Developers gain access to open models that flag multiple harm categories in both prompts and responses with higher precision than previous options.
  • Training mainly on synthetic data still yields models that generalize across different safety-related tasks.
  • The curation pipeline offers a reusable method for creating labeled safety datasets without heavy manual annotation.
  • Releasing the models allows other researchers to build and compare against a new baseline for content moderation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The curation approach might extend to labeling data for related problems such as detecting misinformation or biased outputs.
  • Embedding ShieldGemma-style checks directly into generation loops could lower the rate of unsafe responses in deployed chat systems.
  • Performance gains on internal benchmarks suggest the models could handle domain-specific safety rules if further adapted to particular industries.

Load-bearing premise

The benchmarks used for testing, both public and internal, reflect the kinds of inputs and outputs that appear in actual deployments, and the synthetic data labels remain reliable outside the training distribution.

What would settle it

A substantial drop in AU-PRC scores when the models are tested on a large set of real user conversations and model generations drawn from production systems rather than the current benchmark sets.

read the original abstract

We present ShieldGemma, a comprehensive suite of LLM-based safety content moderation models built upon Gemma2. These models provide robust, state-of-the-art predictions of safety risks across key harm types (sexually explicit, dangerous content, harassment, hate speech) in both user input and LLM-generated output. By evaluating on both public and internal benchmarks, we demonstrate superior performance compared to existing models, such as Llama Guard (+10.8\% AU-PRC on public benchmarks) and WildCard (+4.3\%). Additionally, we present a novel LLM-based data curation pipeline, adaptable to a variety of safety-related tasks and beyond. We have shown strong generalization performance for model trained mainly on synthetic data. By releasing ShieldGemma, we provide a valuable resource to the research community, advancing LLM safety and enabling the creation of more effective content moderation solutions for developers.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper introduces ShieldGemma, a suite of safety content moderation models fine-tuned from Gemma2. It claims state-of-the-art performance in predicting safety risks across harm categories (sexually explicit content, dangerous content, harassment, hate speech) for both user inputs and model outputs. The work relies on a novel LLM-based synthetic data curation pipeline for training data, reports +10.8% AU-PRC gains over Llama Guard and +4.3% over WildCard on public benchmarks, and asserts strong generalization from primarily synthetic training data. Models are released to support community research in LLM safety.

Significance. If the performance margins and generalization claims are substantiated, this would provide a useful open resource for content moderation in generative AI, with the synthetic curation approach potentially adaptable to other safety-related tasks.

major comments (2)
  1. [§4] §4 (Experiments): The central claim of strong generalization and SOTA performance from synthetic data lacks explicit OOD validation. No metrics quantify label agreement with human raters on held-out real-world distributions, nor are distribution-shift measures (e.g., embedding distances or harm-type prevalence shifts) reported between the synthetic training set and the public/internal test sets. This directly undermines the generalization assertion that supports the reported AU-PRC improvements.
  2. [Table 1] Table 1 or main results table: The +10.8% AU-PRC margin over Llama Guard is presented without confidence intervals, statistical significance tests, or details on baseline re-implementations, making it difficult to assess whether the gains are robust or influenced by evaluation protocol choices.
minor comments (2)
  1. [Abstract] Abstract: The mention of 'internal benchmarks' should include a brief description of their construction and any overlap with the synthetic data generation process.
  2. [§3] §3 (Data Curation Pipeline): Provide the exact LLM prompts or model versions used for synthetic label generation to improve reproducibility.

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for their insightful comments, which have helped us improve the presentation and rigor of our work. Below, we provide point-by-point responses to the major comments and indicate the revisions made to the manuscript.

read point-by-point responses

  1. Referee: [§4] §4 (Experiments): The central claim of strong generalization and SOTA performance from synthetic data lacks explicit OOD validation. No metrics quantify label agreement with human raters on held-out real-world distributions, nor are distribution-shift measures (e.g., embedding distances or harm-type prevalence shifts) reported between the synthetic training set and the public/internal test sets. This directly undermines the generalization assertion that supports the reported AU-PRC improvements.

    Authors: We agree that more explicit validation of out-of-distribution generalization would bolster our claims. The public and internal benchmarks are drawn from real-world distributions distinct from our synthetic training data, providing evidence of generalization. To address the referee's concern directly, we have added distribution-shift analyses in the revised §4, including cosine distances between sentence embeddings of synthetic and test data, as well as shifts in harm category prevalence. Regarding human rater agreement, the benchmarks we use already incorporate human annotations where available, but we acknowledge that additional agreement metrics on purely held-out real data would be ideal; we have noted this limitation in the revised manuscript. revision: partial

  2. Referee: [Table 1] Table 1 or main results table: The +10.8% AU-PRC margin over Llama Guard is presented without confidence intervals, statistical significance tests, or details on baseline re-implementations, making it difficult to assess whether the gains are robust or influenced by evaluation protocol choices.

    Authors: We thank the referee for pointing this out. In the revised manuscript, we now include 95% bootstrap confidence intervals for all AU-PRC scores in Table 1. We have also added statistical significance testing using bootstrap resampling to assess the robustness of the performance margins. Furthermore, we have expanded the experimental details in §4 to describe the exact re-implementation of Llama Guard and WildCard, including the prompts and evaluation settings used to ensure fair comparison. revision: yes

standing simulated objections not resolved
  • Additional human annotation for label agreement on held-out real-world data beyond existing benchmark labels.

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper trains ShieldGemma models on synthetic data from a novel LLM curation pipeline and reports performance on separate public and internal benchmarks. No load-bearing step reduces the SOTA claims (+10.8% AU-PRC) or generalization assertions to a self-definition, fitted input renamed as prediction, or self-citation chain. The evaluation metrics are computed directly against external baselines on held-out sets, making the derivation self-contained against independent benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central performance claims rest on the assumption that the chosen public benchmarks are fair and that synthetic labels transfer to real user and model outputs. No new physical constants or mathematical axioms are introduced.

axioms (1)
  • domain assumption Public safety benchmarks are representative of deployment risk distributions
    The abstract compares against Llama Guard and WildCard on these benchmarks without discussing distribution shift or coverage gaps.

pith-pipeline@v0.9.0 · 5720 in / 1287 out tokens · 27927 ms · 2026-05-20T13:12:48.624666+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 22 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Beyond Content: A Comprehensive Speech Toxicity Dataset and Detection Framework Incorporating Paralinguistic Cues

    cs.SD 2026-05 unverdicted novelty 7.0

    ToxiAlert-Bench dataset and dual-head neural network detect toxic speech by distinguishing textual versus paralinguistic sources, reporting 21.1% Macro-F1 and 13% accuracy gains over baselines.

  2. When Alignment Isn't Enough: Response-Path Attacks on LLM Agents

    cs.CR 2026-05 unverdicted novelty 7.0

    A malicious relay can strategically rewrite aligned LLM outputs in BYOK agent architectures to achieve up to 99.1% attack success on benchmarks like AgentDojo and ASB.

  3. Going PLACES: Participatory Localized Red Teaming for Text-to-Image Safety in the Global South

    cs.CY 2026-05 unverdicted novelty 6.0

    A participatory red-teaming project in the Global South created the PLACES dataset of 26k T2I failure examples that reveal unique cultural and linguistic harms missed by existing safety frameworks.

  4. Alignment Dynamics in LLM Fine-Tuning

    cs.LG 2026-05 unverdicted novelty 6.0

    The paper introduces a dynamical model that decomposes alignment updates in LLM fine-tuning into rebound and driving forces and predicts a rehearsal priming effect.

  5. VerifyMAS: Hypothesis Verification for Failure Attribution in LLM Multi-Agent Systems

    cs.CL 2026-05 unverdicted novelty 6.0

    VerifyMAS improves failure attribution in LLM multi-agent systems via hypothesis verification on full trajectories, error taxonomy-based data construction, and fine-tuned verifier models, outperforming prior direct-pr...

  6. LPG: Balancing Efficiency and Policy Reasoning in Latent Policy Guardrails

    cs.CR 2026-05 conditional novelty 6.0

    LPG compresses policy deliberation into 10 latent tokens to reach 84.5% safety accuracy and 11x speedup over explicit reasoning baselines on guardrail benchmarks.

  7. Context-Aware Spear Phishing: Generative AI-Enabled Attacks Against Individuals via Public Social Media Data

    cs.CR 2026-05 conditional novelty 6.0

    Generative AI enables scalable, context-aware spear phishing by extracting profiles from public social media, producing emails that outperform real-world phishing samples in personalization and lower recipient suspicion.

  8. GLiGuard: Schema-Conditioned Classification for LLM Safeguard

    cs.CL 2026-05 unverdicted novelty 6.0

    GLiGuard is a compact schema-conditioned bidirectional encoder that matches 7B-27B guard models on safety benchmarks while delivering up to 16x higher throughput and 17x lower latency.

  9. Revisiting JBShield: Breaking and Rebuilding Representation-Level Jailbreak Defenses

    cs.CR 2026-05 accept novelty 6.0

    JBShield is vulnerable to adaptive JB-GCG attacks (up to 53% ASR) because jailbreak representations occupy a distinct region in refusal-direction space; the new RTV defense using Mahalanobis detection on multi-layer f...

  10. Harmful Intent as a Geometrically Recoverable Feature of LLM Residual Streams

    cs.LG 2026-04 unverdicted novelty 6.0

    Harmful intent is linearly separable in LLM residual streams across 12 models and multiple architectures, reaching mean AUROC 0.982 while showing protocol-dependent directions and strong generalization to held-out har...

  11. Harmful Intent as a Geometrically Recoverable Feature of LLM Residual Streams

    cs.LG 2026-04 unverdicted novelty 6.0

    Harmful intent is geometrically recoverable as a linear direction or angular deviation in LLM residual streams, with high AUROC across 12 models, stable under alignment variants including abliterated ones, and transfe...

  12. LLM Safety From Within: Detecting Harmful Content with Internal Representations

    cs.AI 2026-04 unverdicted novelty 6.0

    SIREN identifies safety neurons via linear probing on internal LLM layers and combines them with adaptive weighting to detect harm, outperforming prior guard models with 250x fewer parameters.

  13. A Lightweight Explainable Guardrail for Prompt Safety

    cs.CL 2026-01 conditional novelty 6.0

    LEG is a compact model that jointly classifies unsafe prompts and explains its decisions using synthetic training data and a custom uncertainty-weighted loss.

  14. Disentangled Safety Adapters Enable Efficient Guardrails and Flexible Inference-Time Alignment

    cs.LG 2025-05 unverdicted novelty 6.0

    Disentangled Safety Adapters decouple safety computations from task-optimized LLMs via lightweight adapters, yielding up to 53% better AUC on safety tasks and dynamic inference-time alignment with reduced performance ...

  15. SafeLens: Deliberate and Efficient Video Guardrails with Fast-and-Slow Screening

    cs.CV 2026-05 unverdicted novelty 5.0

    SafeLens presents a fast-and-slow video guardrail framework that filters the SafeWatch dataset to 2.4% and adds Chain-of-Thought traces to achieve state-of-the-art moderation performance at reduced inference cost.

  16. LiSA: Lifelong Safety Adaptation via Conservative Policy Induction

    cs.LG 2026-05 unverdicted novelty 5.0

    LiSA improves AI guardrails lifelong by inducing conservative policies from sparse noisy failure reports via structured memory, conflict-aware rules, and posterior lower-bound gating.

  17. A Systematic Investigation of The RL-Jailbreaker in LLMs

    cs.LG 2026-05 unverdicted novelty 5.0

    Dense rewards and extended episode lengths in the RL jailbreaking framework are the primary drivers of successful attacks on LLMs.

  18. Guardian-as-an-Advisor: Advancing Next-Generation Guardian Models for Trustworthy LLMs

    cs.LG 2026-04 unverdicted novelty 5.0

    Guardian-as-an-Advisor prepends risk labels and explanations from a guardian model to queries, improving LLM safety compliance and reducing over-refusal while adding minimal compute overhead.

  19. DRAFT: Task Decoupled Latent Reasoning for Agent Safety

    cs.LG 2026-02 unverdicted novelty 5.0

    DRAFT decouples agent safety judgment into latent extraction and reasoning stages, raising average benchmark accuracy from 63.27% to 91.18%.

  20. Bielik Guard: Efficient Polish Language Safety Classifiers for LLM Content Moderation

    cs.CL 2026-02 unverdicted novelty 5.0

    Bielik Guard delivers compact Polish safety classifiers with F1 scores near 0.79 and superior real-prompt precision over baselines.

  21. GLiNER Guard: Unified Encoder Family for Production LLM Safety and Privacy

    cs.CR 2026-05 unverdicted novelty 4.0

    GLiNER Guard provides unified encoder variants for LLM safety and PII detection in a single pass, with high throughput on A100 hardware and a new PII-Bench benchmark.

  22. TWGuard: A Case Study of LLM Safety Guardrails for Localized Linguistic Contexts

    cs.CR 2026-04 unverdicted novelty 4.0

    TWGuard achieves +0.289 F1 improvement and 94.9% false-positive reduction for LLM safety guardrails in the Taiwan linguistic context compared to foundation models and baselines.

Reference graph

Works this paper leans on

22 extracted references · 22 canonical work pages · cited by 21 Pith papers · 9 internal anchors

  1. [1]

    GPT-4 Technical Report

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Al- tenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    J. T. Ash, C. Zhang, A. Krishnamurthy, J. Langford, and A. Agarwal. Deep batch active learning by diverse, uncertaingradientlowerbounds. arXiv preprint arXiv:1906.03671,

    work page arXiv 1906

  3. [3]

    Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learn- ing from human feedback. arXiv preprint arXiv:2204.05862,

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    G. H. Chen, S. Chen, Z. Liu, F. Jiang, and B. Wang. Humans or llms as the judge? a study on judge- ment biases.arXiv preprint arXiv:2402.10669,

    work page arXiv

  5. [5]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint arXiv:1810.04805,

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    J. Gao, R. Pi, Y. Lin, H. Xu, J. Ye, Z. Wu, W. Zhang, X. Liang, Z. Li, and L. Kong. Self-guided noise-free data generation for efficient zero- shot learning.arXiv preprint arXiv:2205.12679,

    work page arXiv

  7. [7]

    Aegis: Online adaptive ai content safety moderation with ensemble of llm experts,

    9 ShieldGemma: Generative AI Content Moderation Based on Gemma S. Ghosh, P. Varshney, E. Galinkin, and C. Parisien. Aegis: Online adaptive ai content safety mod- eration with ensemble of llm experts.arXiv preprint arXiv:2404.05993,

    work page arXiv

  8. [8]

    S. Han, K. Rao, A. Ettinger, L. Jiang, B. Y. Lin, N. Lambert, Y. Choi, and N. Dziri. Wildguard: Openone-stopmoderationtoolsforsafetyrisks, jailbreaks, and refusals of llms.arXiv preprint arXiv:2406.18495,

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    Huang, Y

    H. Huang, Y. Qu, J. Liu, M. Yang, and T. Zhao. An empiricalstudyofllm-as-a-judgeforllmevalua- tion: Fine-tuned judge models are task-specific classifiers. arXiv preprint arXiv:2403.02839,

    work page arXiv

  10. [10]

    H. Inan, K. Upasani, J. Chi, R. Rungta, K. Iyer, Y. Mao, M. Tontchev, Q. Hu, B. Fuller, D. Tes- tuggine, et al. Llama guard: Llm-based input- output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674,

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    URL https://arxiv.org/abs/2307.04657. S. Y. Kim, H. Park, K. Shin, and K.-M. Kim. Ask me what you need: Product retrieval us- ing knowledge from gpt-3. arXiv preprint arXiv:2207.02516,

    work page arXiv

  12. [12]

    Kurakin, N

    A. Kurakin, N. Ponomareva, U. Syed, L. Mac- Dermed, and A. Terzis. Harnessing large- language models to generate private synthetic text. arXiv preprint arXiv:2306.01684,

    work page arXiv

  13. [13]

    L. Li, B. Dong, R. Wang, X. Hu, W. Zuo, D. Lin, Y. Qiao, and J. Shao. Salad-bench: A hierar- chical and comprehensive safety benchmark for large language models. arXiv preprint arXiv:2402.05044,

    work page arXiv

  14. [14]

    URL https://arxiv.org/abs/2310.17389. N. Liu, L. Chen, X. Tian, W. Zou, K. Chen, and M. Cui. From llm to conversational agent: A memory enhanced architecture with fine- tuning of large language models.arXiv preprint arXiv:2401.02777,

    work page arXiv

  15. [15]

    L. Long, R. Wang, R. Xiao, J. Zhao, X. Ding, G.Chen,andH.Wang. Onllms-drivensynthetic data generation, curation, and evaluation: A survey. arXiv preprint arXiv:2406.15126,

    work page arXiv

  16. [16]

    URL https://arxiv.org/abs/2402.04249. M. Nasr, N. Carlini, J. Hayase, M. Jagielski, A. F. Cooper, D. Ippolito, C. A. Choquette-Choo, E. Wallace, F. Tramèr, and K. Lee. Scal- able extraction of training data from (pro- duction) language models. arXiv preprint arXiv:2311.17035,

    work page internal anchor Pith review Pith/arXiv arXiv

  17. [17]

    Radharapu, K

    B. Radharapu, K. Robinson, L. Aroyo, and P. La- hoti. Aart: Ai-assistedred-teamingwithdiverse data generation for new llm-powered applica- tions. arXiv preprint arXiv:2311.08592,

    work page arXiv

  18. [18]

    G. Sahu, P. Rodriguez, I. H. Laradji, P. Atighe- hchian, D. Vazquez, and D. Bahdanau. Data augmentation for intent classification with off- the-shelf large language models.arXiv preprint arXiv:2204.01959,

    work page arXiv

  19. [19]

    Active Learning for Convolutional Neural Networks: A Core-Set Approach

    10 ShieldGemma: Generative AI Content Moderation Based on Gemma O. Sener and S. Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489,

    work page internal anchor Pith review Pith/arXiv arXiv

  20. [20]

    i’m sorry to hear that

    E. M. Smith, M. Hall, M. Kambadur, E. Presani, and A. Williams. " i’m sorry to hear that": Finding new biases in language models with a holistic descriptor dataset. arXiv preprint arXiv:2205.09209,

    work page arXiv

  21. [21]

    G. Team. Gemma. 2024a. doi: 10.34740/ KAGGLE/M/3301. URL https://www.kaggle. com/m/3301. G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805,

    work page internal anchor Pith review Pith/arXiv arXiv

  22. [22]

    G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, J. Love, et al. Gemma: Open mod- els based on gemini research and technology. arXiv preprint arXiv:2403.08295,

    work page internal anchor Pith review Pith/arXiv arXiv