Pith. sign in

REVIEW 3 major objections 6 minor 13 cited by

Granite Guardian

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Granite Guardian claims that a single open-source guardrail model can detect harmful content and RAG hallucinations at deployment-competitive accuracy, reporting an aggregated AUC of 0.871 on harm benchmarks and 0.854 on groundedness…

desk verdict A genuinely useful open guardrail model release, but the headline harm AUC is threatened by possible HH-RLHF train/test overlap that the paper never rules out. read the letter →

arxiv 2412.07724 v2 pith:QDCTYHPL submitted 2024-12-10 cs.CL

Inkit Padhi , Manish Nagireddy , Giandomenico Cornacchia , Subhajit Chaudhury , Tejaswini Pedapati , Pierre Dognin , Keerthiram Murugesan , Erik Miehling
show 15 more authors
Martín Santillán Cooper Kieran Fraser Giulio Zizzo Muhammad Zaid Hameed Mark Purcell Michael Desmond Qian Pan Zahra Ashktorab Inge Vejsbjerg Elizabeth M. Daly Michael Hind Werner Geyer Ambrish Rawat Kush R. Varshney Prasanna Sattigeri
This is my paper · ORCID
classification cs.CL
keywords riskdetectionLLMguardrailsharmfulcontentjailbreakRAGhallucinationgroundednesssafetyinstructiontemplatesupervisedfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces Granite Guardian, a pair of open-source language models (2B and 8B parameters) meant to act as a single guardrail for any LLM pipeline. The authors claim these models detect both classic harmful content—an umbrella 'harm' category plus social bias, profanity, violence, sexual content, unethical behavior, and jailbreaking—and RAG-specific hallucination risks: context relevance, groundedness, and answer relevance. On aggregated public benchmarks, the 8B model reports an AUC of 0.871 for harmful-content detection, ahead of the Llama Guard and ShieldGemma families, and an average AUC of 0.854 on the TRUE groundedness benchmark, making it the best-performing fully open-source groundedness checker. The value of the claim is that one model, released openly, can replace separate content-moderation and fact-consistency components.

What carries the argument

The mechanism that carries the argument is a safety instruction template: a prompt that names the target span (user message, assistant message, or context), inserts the text between control tokens, and applies a plain-language risk definition, instructing the model to answer 'Yes' or 'No'. The probability of risk is computed from the top-$k$ token log-likelihoods, aggregating all tokens that contain 'Yes' or 'No' as $\text{score}_{\text{unsafe}} = \sum_{u \in U|k} \exp(\mathrm{LL}(u))$ and softmax-normalizing the pair; with $k=20$ this yields a confidence score from the same checkpoint for any risk definition. The template allows custom risk definitions at inference time, and the synthetic-data pipeline supplies the hard cases: 24 jailbreak revision strategies and RAG negatives seeded from HotPotQA, SQuAD2, MNLI, and SNLI.

What would settle it

Compare the released Granite Guardian training data against the test splits of AegisSafetyTest, BeaverTails, and SafeRLHF using exact-match and near-duplicate retrieval; if a material fraction of the benchmark examples appears in the training set, the aggregated AUC of 0.871 reflects memorization and the generalization claim fails.

Watch

Extended reading notes

Core claim

The central claim is that a single supervised fine-tuned model family can detect safety risks across both the input/output moderation axis and the retrieval-augmented generation quality axis at deployment-competitive accuracy. Granite Guardian is derived from Granite 3.0 instruct checkpoints and trained on human-annotated HH-RLHF first-turn prompts paired with model responses, plus synthetic data for contrastive benign/harmful prompts, jailbreak revisions, and RAG negative samples. On the aggregated harm benchmarks, Granite-Guardian-3.0-8B achieves the best AUC (0.871), AUPRC (0.846), and F1 (0.758) among compared models; on the TRUE datasets it averages 0.854 AUC for groundedness, placing second behind MiniCheck and first among fully open-source models. The paper also reports that with the jailbreak risk definition the 8B model reaches a recall of 1.0 on ToxicChat jailbreak prompts.

Load-bearing premise

The headline scores assume that Granite Guardian's own training data does not overlap with the benchmark examples used to test it, but several test sets (AegisSafetyTest, BeaverTails, SafeRLHF) and the training prompts share the same HH-RLHF source, and the paper does not show that these overlaps were removed.

Editorial extensions

If this is right

  • Deployments can use one open model instead of separate toxic-content and fact-consistency checkers, since Granite Guardian covers both with a single inference call.
  • Operators can set thresholds on the computed risk probability to tune for low false-positive rates; at FPr 0.1 the 8B model reaches TPr 0.68 on the aggregated harm benchmark.
  • The 2B variant offers a resource-constrained option with aggregate AUC 0.782 and recall 0.747, making safety filtering feasible on smaller infrastructure.
  • Because the safety template accepts new risk definitions, enterprises can extend the same checkpoint to custom policies without retraining, although the paper advises testing such extensions.
  • Open release of the models lets practitioners audit and adapt the guardrail, supporting the paper's stated goal of shared responsible-AI infrastructure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported margin over Llama Guard and ShieldGemma may shrink once the HH-RLHF ancestry of both training and benchmark sets is accounted for; a decontamination study would settle how much of the 0.871 AUC is generalization.
  • The top-k yes/no scoring trick is model-agnostic: the same template could be applied to other instruction-tuned base models, so the approach may be a recipe for building guardrails rather than a one-off model.
  • A direct test of the jailbreak claim would be to run the model against a fresh, never-published red-teaming set; recall 1.0 on ToxicChat's jailbreak subset does not by itself show robustness to unseen attack strategies.
  • The RAG half of the claim could be extended from groundedness to the full triad—context relevance and answer relevance—on held-out enterprise data, which the paper does not benchmark.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces Granite Guardian, a suite of 2B and 8B safeguard models fine-tuned from Granite 3.0 instruct models, designed to detect harmful content (harm, social bias, profanity, violence, sexual content, unethical behavior, jailbreaking) and RAG-specific risks (context relevance, groundedness, answer relevance). The training data combines human annotations on HH-RLHF-derived prompts and responses, synthetic contrastive and jailbreak data, and synthetic RAG data generated from HotPotQA, SQuAD2, MNLI, and SNLI. The authors evaluate the models on public harmfulness benchmarks and TRUE groundedness benchmarks, reporting an aggregate harm AUC of 0.871 for the 8B model and an average groundedness AUC of 0.854, and comparing against Llama Guard, ShieldGemma, ANLI, WeCheck, and MiniCheck.

Significance. If the reported results are robust, the paper makes a useful contribution: an open-source, unified guardrail covering both content-safety and RAG hallucination risks, with publicly released model weights, cookbooks, and a flexible safety-instruction template that supports custom risk definitions. The use of multiple human annotators per sample, reported inter-annotator agreement, and evaluation across low false-positive-rate operating points are strengths. However, the central generalization claim is currently not established because the paper does not demonstrate that the training data are disjoint from several of the key evaluation benchmarks, and the abstract's 'state-of-the-art' wording is stronger than the paper's own tables support. These issues are repairable and do not invalidate the underlying model development, but they must be addressed before the headline claims can be accepted.

major comments (3)
  1. [§3.1, §5.3, Table 4, §6.1] The paper does not establish that Granite Guardian's own training data are disjoint from the evaluation benchmarks, although several benchmarks are derived from the same source as the training data. Section 3.1 states that the first annotation phase used 7,000 first-turn prompts from HH-RLHF and the second phase selected 1,000 additional HH-RLHF-derived examples, while Section 5.3 and Table 4 describe AegisSafetyTest as 'derived from Anthropic's HH-RLHF harmlessness dataset', BeaverTails prompts as 'derived from HH-RLHF red teaming', and SafeRLHF as including BeaverTails prompts. The only statement about overlap, in Section 6.1, says 'The evaluations results do not consider train-test overlap for baselines', which addresses the baselines rather than Granite Guardian. Because the headline harm AUC of 0.871 is computed on exactly these datasets, the authors should report a formal decontamination analysis (for example, exact and near-duplicate overlap of training and evaluation prompts and responses, with explicit thresholds), retrain without the overlapping sources, or re-state the claims as in-distribution performance. This is load-bearing for the abstract's generalization claim.
  2. [§6.2, Table 8; Abstract; §1] The claims that Granite Guardian 'outperforms other open- and closed-source models' and is 'the most generalizable and competitive model available' are not supported by the paper's own results on the RAG benchmarks. In Table 8, Llama-3.1-Bespoke-MiniCheck-7B achieves an average AUC of 0.873 on the TRUE datasets, which is higher than Granite-Guardian-3.0-8B's 0.854, and the text in Section 6.2 correctly states that the 8B model 'ranks second on average AUC' and is 'the best-performing fully open-source model'. The abstract and introduction should be revised to report the comparative ranking accurately, and the statement about closed-source models should be removed or accompanied by closed-source baselines, since Section 5.2 lists no closed-source systems.
  3. [§5.3, §6.1] The evaluation section says the selected benchmarks 'prioritize out-of-distribution and public datasets' (Section 5.3), but the HH-RLHF-derived benchmarks are not out-of-distribution relative to the training data described in Section 3.1. Even without verbatim overlap, the distribution of prompt topics and styles in AegisSafetyTest, BeaverTails, and SafeRLHF is substantially represented in the training sample, so the aggregated AUC partly reflects distributional familiarity rather than pure out-of-distribution generalization. To support the 'most generalizable' claim, the authors should report performance separately on benchmarks whose source distributions are disjoint from training, such as ToxicChat, OpenAI Moderation, SimpleSafetyTests, and HarmBench, and discuss any performance differences.
minor comments (6)
  1. [Table 8] The column headers MNBN, QX, QC, SumE, and DialF do not match the dataset names used in Section 5.3 (MNBM, QAGS-XSUM, QAGS-CNN/DM, SummEval, and DialFact, respectively); please align the table with the text.
  2. [§2.1.1] 'This category is applies to both prompts and responses' should read 'This category applies to both prompts and responses.'
  3. [§5.3] 'any instance in these datasets that have have been marked' contains a duplicated 'have' that should be removed.
  4. [§4.3] The notation U|k and S|k should be defined more precisely: 'top-k tokens' should specify top-k by log-likelihood and whether the selection is restricted to the first generated token position before the softmax normalization.
  5. [§7.2] The limitations section does not mention the train-evaluation overlap issue; a sentence acknowledging the need for decontamination analysis in future releases would improve transparency.
  6. [§4.2] The training setup reports a learning rate, optimizer, and epoch count, but the dataset size and the validation split used for checkpoint selection are not stated; please provide these details for reproducibility.

Circularity Check

1 steps flagged · score 5.0 of 10

Partial data-source circularity: the headline harm AUC is measured on benchmarks derived from the same HH-RLHF source used to construct Granite Guardian's training set, with no reported decontamination.

  1. fitted input called prediction [Section 3.1 (training data) vs. Section 5.3 and Section 6.1 (benchmarks and results)]
    ""The first phase focused on samples from human preference data on harmlessness - HH-RLHF (Bai et al., 2022). Specifically, only the first turn (containing the human prompt) was selected ... This process produced 7,000 unique (prompt, response) pairs for annotation." "AegisSafetyTest ... consists of 1,199 entries from Anthropic's HH-RLHF harmlessness dataset." "BeaverTails ... The prompts are derived from HH-RLHF red teaming and Sun et al. (2023)." "SafeRLHF ... includes prompts of the BeaverTails dataset." "The evaluations results do not consider train-test overlap for baselines.""

    Granite Guardian's harm-detection training set is built from HH-RLHF: 7,000 first-turn prompts in phase 1, plus 1,000 more HH-RLHF examples selected by uncertainty sampling in phase 2. The aggregated harm benchmark behind the headline AUC of 0.871 includes AegisSafetyTest, which the paper itself states is composed of entries from Anthropic's HH-RLHF harmlessness dataset, and BeaverTails and SafeRLHF, whose prompts come from HH-RLHF red teaming or from BeaverTails. The paper never reports removing HH-RLHF-sourced instances from these test sets; the only overlap statement says evaluations "do not consider train-test overlap for baselines," which does not cover Granite Guardian's own training data.

full rationale

Granite Guardian's headline harm-detection result is an evaluation claim rather than a formal derivation, and most of the pipeline (synthetic jailbreak and RAG data, safety template, scoring rule) is not circular. The circular element is confined to the harm benchmark suite: training draws 7,000 plus 1,000 examples from HH-RLHF in Section 3.1, while three of the eight aggregated test datasets are themselves HH-RLHF-derived (AegisSafetyTest, BeaverTails, SafeRLHF) in Section 5.3. The paper's only overlap caveat, in Section 6.1, disclaims overlap for baselines and therefore does not establish that Granite Guardian's own training inputs were removed from evaluation. Consequently, the quoted AUC of 0.871 is partially an in-sample memorization score for the overlapping subsets, not an independent out-of-distribution prediction. This is a partial, data-source circularity; it does not collapse the whole paper, because the remaining harm benchmarks (ToxicChat, OpenAI Moderation, SimpleSafetyTests, HarmBench, XSTEST) and the RAG groundedness result (AUC 0.854 on TRUE, trained from HotPotQA, SQuAD, MNLI, and SNLI seeds) are not similarly coupled to the training source.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on evaluation choices and data assumptions rather than on derived parameters. The key burden is the assumption that public benchmark labels are reliable and that HH-RLHF-derived test sets do not overlap the training data. No invented entities are introduced.

free parameters (4)
  • top-k in risk score aggregation = 20
    Eq. (1) sums exp(log-likelihood) over top-k tokens containing 'Yes' or 'No'; k=20 is chosen by hand in Section 4.3.
  • metric threshold for F1/precision/recall = 0.5
    Section 5.1 computes threshold-based metrics at a fixed threshold of 0.5; this affects reported F1 but not AUC.
  • benchmark label mapping = binary harm from each dataset's own safety labels
    Section 5.3 assigns a positive/umbrella harmful label to any instance marked unsafe under its own taxonomy; this mapping determines the ground truth of the aggregated evaluation.
  • training hyperparameters = learning rate 1e-6, up to 7 epochs, 5 gradient accumulation steps
    Section 4.2 specifies these SFT choices and selects the checkpoint by minimum validation loss; they shape the final model but are not fitted to the evaluation benchmarks.
assumptions (4)
  • domain assumption Human annotation quality and demographic diversity produce reliable labels.
    Section 3.1 relies on three annotators per sample and reports inter-annotator agreements, but no adjudication or gold-standard validation is described.
  • domain assumption Public benchmark labels are accurate ground truth for harmfulness and groundedness.
    Section 5.3 maps each dataset's own labels to binary harm or groundedness without auditing label noise.
  • domain assumption Synthetic data generated by Mixtral and Granite models is representative of real-world harmful, jailbreak, and RAG content.
    Section 3.2 uses LLM-generated prompts and responses as training data; there is no external validation that this distribution matches deployment.
  • domain assumption The Granite 3.0 base models provide a suitable foundation for safety fine-tuning.
    Section 4.2 fine-tunes Granite 3.0 instruct variants; the report does not compare with other base models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Granite Guardian." pith.science (2026). https://pith.science/paper/QDCTYHPL

@misc{pith2026241207724,
  author       = {Pith},
  title        = {Pith review of: Granite Guardian},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QDCTYHPL}},
  note         = {Machine review of arXiv:2412.07724}
}
read the original abstract

We introduce the Granite Guardian models, a suite of safeguards designed to provide risk detection for prompts and responses, enabling safe and responsible use in combination with any large language model (LLM). These models offer comprehensive coverage across multiple risk dimensions, including social bias, profanity, violence, sexual content, unethical behavior, jailbreaking, and hallucination-related risks such as context relevance, groundedness, and answer relevance for retrieval-augmented generation (RAG). Trained on a unique dataset combining human annotations from diverse sources and synthetic data, Granite Guardian models address risks typically overlooked by traditional risk detection models, such as jailbreaks and RAG-specific issues. With AUC scores of 0.871 and 0.854 on harmful content and RAG-hallucination-related benchmarks respectively, Granite Guardian is the most generalizable and competitive model available in the space. Released as open-source, Granite Guardian aims to promote responsible AI development across the community. https://github.com/ibm-granite/granite-guardian

Figures

Figures reproduced from arXiv: 2412.07724 by the authors.

Figure 1
Figure 1. Annotation guidelines 3.2 Synthetic data The training data was augmented with synthetic samples to address three key areas: (1) complex examples that highlight the contrast between benign and harmful prompts, (2) prompts containing adversarial components that pose a risk of jailbreaks, and (3) data designed to enhance risk detection in RAG use cases. The following sections provide a detailed explanation of each area… view at source ↗
Figure 2
Figure 2. Prompt for benign prompt generation Using the prompt in [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Prompt for typical harmful prompt generation [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Prompt for adversarial harmful prompt generation [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Example revision strategy for adversarial prompt transformation [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Prompt for RAG synthetic data generation [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: (Left) Safety instruction template parameterized for detecting risks associated with [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Comparison of ROC curves for 2B (left) and 8B (right) Granite Guardian versions. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: The bar chart plot presents the macro F1 scores for Granite Guardian models [PITH_FULL_IMAGE:figures/full_fig_p030_9.png]
Figure 10
Figure 10. Figure 10: The bar chart plot presents the F1 scores for the Granite Guardian models against [PITH_FULL_IMAGE:figures/full_fig_p031_10.png]
Figure 11
Figure 11. Figure 11: The bar chart plot presents the TPr for the Granite Guardian models against [PITH_FULL_IMAGE:figures/full_fig_p031_11.png]
Figure 12
Figure 12. Figure 12: The bar chart plot presents the FPr for the Granite Guardian models against [PITH_FULL_IMAGE:figures/full_fig_p031_12.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 13 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Symbolic Augmentation Closes a Canonical-Equivalence Blind Spot in Neural Fact-Checkers

    cs.AI 2026-05 conditional novelty 7.0 of 10

    Typed quantity verification exposes a canonical-equivalence blind spot in neural fact-checkers; Symbolic Augmentation fixes it (36.5%→98.2%) and transfers to SciFact-Open (+0.037 binary macro-F1).

  2. JANUS: Foreseeing Latent Risk for Long-Horizon Agent Safety

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A guard trained to anticipate safety-relevant futures from partial trajectories cuts average attack success from 23.0% to 7.1% across four agent-safety benchmarks.

  3. DARWIN: Evolving Jailbreak Adversary and Guardrail for LLM Safety Evaluation and Protection

    cs.CR 2026-07 conditional novelty 6.0 of 10

    An evolving attack-defense loop, DARWIN, achieves state-of-the-art jailbreak success rates on frontier LLMs/guardrails and trains a guardrail with 91.6% average unsafe recall while retaining ~100% benign pass rate.

  4. CPInj: Uncovering Prompt Injection Risks in Textual Collaborative Prompt Optimization

    cs.CR 2026-07 conditional novelty 6.0 of 10

    CPInj demonstrates that federated textual prompt optimization (a TextGrad-style loop) is vulnerable to a multi-objective injection attack that persists through aggregation, degrades accuracy by up to 55 points, and ou...

  5. HyperSafe: Inference-Time Safety Recovery for Fine-Tuned Language Models

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A hypernetwork maps layer-wise activation fingerprints of a fine-tuned LLM to a Safe Side Network that routes harmful prompts to refusal without editing model weights.

  6. Adversarial Bug Reports as a Security Risk in Language Model-Based Automated Program Repair

    cs.SE 2025-09 conditional novelty 6.0 of 10

    Adversarial bug reports induced attacker-desired patches in 90% of trials, while the best tested pre-repair filter caught only 47%, exposing a structural weakness in LLM-based automated program repair.

  7. Concealment of Intent: A Game-Theoretic Analysis

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Intent-hiding adversarial prompting that mixes malicious intents with innocuous skills bypasses prompt and response filters, and a game-theoretic analysis quantifies the attacker's scaling advantage.

  8. An Annotated Reading of 'The Singer of Tales' in the LLM Era

    cs.CY 2025-02 conditional novelty 6.0 of 10

    LLM generation resembles oral-formulaic composition: single-pass, pattern-based, and non-authorial, so AI output should be treated as a new post-literate medium.

  9. Benchmarking Open-Source Safety Guard Models: A Comprehensive Evaluation

    cs.CL 2026-04 conditional novelty 5.5 of 10

    Qwen Guard (4B) reaches 83.97% recall on a 79k NIST-aligned safety benchmark while larger models such as Llama Guard 12B and GPT-OSS 20B miss up to 75% of unsafe content; model size does not predict detection performance.

  10. Semalith v1.4: A Calibrated 184M Safety Classifier Achieving State-of-the-Art Prompt-Injection Detection at 44x Fewer Parameters than Llama-Guard-3-8B

    cs.LG 2026-05 reject novelty 5.0 of 10

    A 184M-parameter DeBERTa-v3 fine-tuned model is claimed to beat Llama-Guard-3-8B on all tested prompt-injection benchmarks while adding BFSI regulatory labels, but a leaked training/eval overlap undermines the zero-FPR claim.

  11. SafeTy Reasoning Elicitation Alignment for Multi-Turn Dialogues

    cs.CL 2025-05 conditional novelty 5.0 of 10

    STREAM fine-tunes a small reasoning model on human-labeled, reason-annotated multi-turn dialogues and uses it to warn target LLMs, cutting average attack success rates by roughly half while keeping benchmark scores close.

  12. JavelinGuard: Low-Cost Transformer Architectures for LLM Security

    cs.LG 2025-06 reject novelty 4.0 of 10

    A study of five small transformer classifier architectures for LLM jailbreak and prompt injection detection claims low-latency accuracy comparable to large models, led by the multi-task Raudra design.

  13. Dark LLMs: The Growing Threat of Unaligned AI Models

    cs.CL 2025-05 reject novelty 2.0 of 10

    The paper is a safety warning essay that asserts a universal jailbreak still works on many commercial LLMs without disclosing any measurement or methodology.

Reference graph

Works this paper leans on

37 extracted references · 16 canonical work pages · cited by 13 Pith papers

  1. [1]

    Berger, Bishwaranjan Bhattacharjee, Djallel Bouneffouf, Subhajit Chaudhury, Pin-Yu Chen, Lamogha Chiazor, Elizabeth M

    Swapnaja Achintalwar, Adriana Alvarado Garcia, Ateret Anaby-Tavor, Ioana Baldini, Sara E. Berger, Bishwaranjan Bhattacharjee, Djallel Bouneffouf, Subhajit Chaudhury, Pin-Yu Chen, Lamogha Chiazor, Elizabeth M. Daly, Rog´erio Abreu de Paula, Pierre L. Dognin, Eitan Farchi, Soumya Ghosh, Michael Hind, Raya Horesh, George Kour, Ja Young Lee, Erik Miehling, Ke...

    arXiv

  2. [3]

    org/CorpusID:269430991

    URL https://api.semanticscholar. org/CorpusID:269430991. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kada- vath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, ...

    arXiv

  3. [7]

    Aegis: Online adaptive ai content safety moderation with ensemble of llm experts

    Shaona Ghosh, Prasoon Varshney, Erick Galinkin, and Christopher Parisien. Aegis: Online adaptive ai content safety moderation with ensemble of llm experts. arXiv preprint arXiv:2404.05993,

    arXiv

  4. [8]

    DialFact: A Benchmark for Fact-Checking in Dialogue

    Prakhar Gupta, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. Dialfact: A benchmark for fact-checking in dialogue. arXiv preprint arXiv:2110.08222,

    work page Pith review arXiv

  5. [9]

    Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms

    Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms. CoRR, abs/2406.18495,

    arXiv

  6. [11]

    URL https:// aclanthology.org/2021.emnlp-main.619

    Association for Computational Linguistics. URL https:// aclanthology.org/2021.emnlp-main.619. Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. True: Re-evaluating factual consistency evaluation. In Proceedings of the 2022 Conference of the North ...

    work page 2021

  7. [12]

    Llama guard: Llm-based input-output safeguard for human-ai conversations

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama guard: Llm-based input-output safeguard for human-ai conversations. CoRR, abs/2312.06674,

    arXiv

  8. [14]

    Wildteam- ing at scale: From in-the-wild jailbreaks to (adversarially) safer language models

    Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, and Nouha Dziri. Wildteam- ing at scale: From in-the-wild jailbreaks to (adversarially) safer language models. CoRR, abs/2406.18510,

    arXiv

Show all 37 references
  1. [15]

    Toxicchat: Unveiling hidden challenges of toxicity detection in real-world user-ai conversation

    Zi Lin, Zihan Wang, Yongqi Tong, Yangkun Wang, Yuxin Guo, Yujia Wang, and Jingbo Shang. Toxicchat: Unveiling hidden challenges of toxicity detection in real-world user-ai conversation. In The 2023 Conference on Empirical Methods in Natural Language Processing,

    2023

  2. [16]

    On faithfulness and factuality in abstractive summarization

    Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pp. 1906–1919, Online, July

    1906

  3. [18]

    AI safety v0.5 proof of concept

    MLCommons. AI safety v0.5 proof of concept. https://mlcommons.org/2024/04/ mlc-aisafety-v0-5-poc/ . Manish Nagireddy, Lamogha Chiazor, Moninder Singh, and Ioana Baldini. SocialStigmaQA: A benchmark to uncover stigma amplification in generative language models. In AAAI, pp. 214...

    2024

  4. [19]

    Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization

    23 IBM Granite Guardian Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing...

    2018

  5. [20]

    OWASP Top 10 for Large Language Model Applications

    OWASP. OWASP Top 10 for Large Language Model Applications. https://genai.owasp. org/resource/owasp-top-10-for-llm-applications-2025/ ,

    2025

  6. [21]

    URL https://doi.org/10.1177/0146167217741313

    doi: 10.1177/0146167217741313. URL https://doi.org/10.1177/0146167217741313. PMID: 29290150. Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. Understanding factual- ity in abstractive summarization with FRANK: A benchmark for factuality metrics. In Proceedings of t...

    doi

  7. [22]

    doi: 10.18653/v1/2021.naacl-main.383

    Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.383. URL https://aclanthology.org/2021.naacl-main.383. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits...

    2021 doi

  8. [23]

    doi: 10.18653/v1/P18-2124

    Association for Computational Linguistics. doi: 10.18653/v1/P18-2124. URL https://aclanthology.org/P18-2124. Ambrish Rawat, Stefan Schoepf, Giulio Zizzo, Giandomenico Cornacchia, Muhammad Zaid Hameed, Kieran Fraser, Erik Miehling, Beat Buesser, Elizabeth M. Daly, Mark Purcell,...

    doi

  9. [24]

    Paul R¨ottger, Hannah Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy

    URL https://arxiv.org/ abs/2409.15398. Paul R¨ottger, Hannah Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. In NAACL-HLT, pp. 5377–5400. Association for Comp...

    arXiv

  10. [25]

    Boyd-Graber

    Sander Schulhoff, Jeremy Pinto, Anaum Khan, Louis-Fran c ¸ois Bouchard, Chenglei Si, Svetlina Anati, Valen Tagliabue, Anson Liu Kost, Christopher Carnahan, and Jordan L. Boyd-Graber. Ignore this title and hackaprompt: Exposing systemic vulnerabilities of llms through a global ...

    arXiv

  11. [26]

    URLhttps://doi.org/10.48550/arXiv.2311.16119

    doi: 10.48550/ARXIV .2311.16119. URLhttps://doi.org/10.48550/arXiv.2311.16119. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. ”do anything now”: Characterizing and evaluating in-the-wild jailbreak prompts on large language models. CoRR, abs/2308.03825,

    doi arXiv

  12. [27]

    24 IBM Granite Guardian Hao Sun, Zhexin Zhang, Jiawen Deng, Jiale Cheng, and Minlie Huang

    URL https://arxiv.org/abs/2408.12622. 24 IBM Granite Guardian Hao Sun, Zhexin Zhang, Jiawen Deng, Jiale Cheng, and Minlie Huang. Safety assessment of chinese large language models. CoRR, abs/2304.10436,

    arXiv

  13. [28]

    MiniCheck: Efficient fact-checking of LLMs on grounding documents

    Liyan Tang, Philippe Laban, and Greg Durrett. MiniCheck: Efficient fact-checking of LLMs on grounding documents. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 8818–8847, ...

    2024

  14. [29]

    URL https://aclanthology.org/2024.emnlp-main.499

    Association for Computational Linguistics. URL https://aclanthology.org/2024.emnlp-main.499. Christoph Tillmann, Aashka Trivedi, Sara Rosenthal, Santosh Borse, Rong Zhang, Avirup Sil, and Bishwaranjan Bhattacharjee. Muted: Multilingual targeted offensive speech identifi- catio...

    2024

  15. [30]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    arXiv

  16. [31]

    Rag triad

    TruLens. Rag triad. URL https://www.trulens.org/getting started/core concepts/rag triad. Bertie Vidgen, Hannah Rose Kirk, Rebecca Qian, Nino Scherrer, Anand Kannappan, Scott A Hale, and Paul R¨ottger. Simplesafetytests: a test suite for identifying critical safety risks in lar...

    arXiv

  17. [32]

    doi: 10.18653/v1/2020.acl-main.450

    As- sociation for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.450. URL https://aclanthology.org/2020.acl-main.450. Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. Do-not- answer: Evaluating safeguards in llms. In EACL (Findings), pp. 896–91...

    2020 doi

  18. [33]

    A broad-coverage challenge corpus for sentence understanding through inference

    Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In 2018 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2018,...

    2018

  19. [34]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhut- dinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii (eds.), Proc...

    2018

  20. [35]

    doi: 10.18653/v1/D18-1259

    As- sociation for Computational Linguistics. doi: 10.18653/v1/D18-1259. URL https: //aclanthology.org/D18-1259. 25 IBM Granite Guardian Wenjun Zeng, Yuchi Liu, Ryan Mullins, Ludovic Peran, Joe Fernandez, Hamza Harkous, Karthik Narasimhan, Drew Proud, Piyush Kumar, Bhaktipriya ...

    arXiv

  21. [36]

    PAWS: Paraphrase adversaries from word scrambling

    Yuan Zhang, Jason Baldridge, and Luheng He. PAWS: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , ...

    2019

  22. [2015]

    q2: Evaluating factual consistency in knowledge-grounded dialogues via question generation and question answering

    Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. q2: Evaluating factual consistency in knowledge-grounded dialogues via question generation and question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Langu...

    2021

  23. [2018]

    Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang

    URL https://datasociety.net/ library/content-or-context-moderation/. Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: safe reinforcement learning from human feedback. In The Twelfth International Conference on Lear...

    2024

  24. [2019]

    doi: 10.18653/v1/N19-1131

    Association for Computational Linguistics. doi: 10.18653/v1/N19-1131. URL https://aclanthology.org/ N19-1131. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. CoRR, abs/2307.15043,

    arXiv

  25. [2020]

    doi: 10.18653/v1/2020.acl-main.173

    As- sociation for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.173. URL https://aclanthology.org/2020.acl-main.173. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan He...

    2020 doi

  26. [2021]

    Latent hatred: A benchmark for understanding implicit hate speech

    Mai ElSherief, Caleb Ziems, David Muchlinski, Vaishnavi Anupindi, Jordyn Seybolt, Mun- mun De Choudhury, and Diyi Yang. Latent hatred: A benchmark for understanding implicit hate speech. CoRR, abs/2109.05322,

    arXiv

  27. [2022]

    A large annotated corpus for learning natural language inference

    Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 632–642,

    2015

  28. [2023]

    URL https://doi.org/10.48550/ arXiv.2312.06674

    doi: 10.48550/ARXIV .2312.06674. URL https://doi.org/10.48550/ arXiv.2312.06674. Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of LLM via a human-prefer...

    doi arXiv

  29. [2024]

    Evaluations of machine learning privacy defenses are misleading

    Michael Aerni, Jie Zhang, and Florian Tram `er. Evaluations of machine learning privacy defenses are misleading. ArXiv, abs/2404.17399,

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.