Beyond the Safety Bundle: Auditing the Helpful and Harmless Dataset

Golnoosh Farnadi; Jackie CK Cheung; Jonathan Cola\c{c}o Carr; Khaoula Chehbouni; Yash More

REVIEW 4 major objections 4 minor 13 references

Beyond the Safety Bundle: Auditing the Helpful and Harmless Dataset

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Auditing the Helpful and Harmless dataset shows that its "harmless" responses are mostly low quality, and models trained on it learn to refuse safe prompts about demographic groups more often, creating disparate safety behavior.

desk verdict First real audit of the HH preference dataset, with striking descriptive findings; the causal claim about disparate safety behaviors rests on a confounded experiment and should be softened. read the letter →

arxiv 2411.08243 v3 pith:SAR3ZRQV submitted 2024-11-12 cs.CL cs.CY

Khaoula Chehbouni , Jonathan Colaço Carr , Yash More , Jackie CK Cheung , Golnoosh Farnadi This is my paper · ORCID

classification cs.CLcs.CY

keywords datasetaudithelpfulharmlesshumanpreferencedatasafetyalignmentexaggeratedbehaviordemographicdisparitydirectoptimizationrefusal

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper audits the Helpful and Harmless (HH) dataset, a widely used human-preference dataset for steering language models toward safe behavior. It finds that the dataset's chosen "harmless" responses are rarely acceptable: only about 12 percent truly refuse harm, while roughly 44 percent are unhelpful failures and 44 percent are themselves harmful. Training three base models with direct preference optimization on the full dataset makes them refuse safe prompts that mention demographic identity terms at far higher rates than models trained without identity-containing samples, and these refusal rates track the overrepresentation of identity terms in the harmless split. The paper argues that the dataset thereby teaches models to associate demographic terms with harmfulness, producing disparate safety behavior across groups. This matters because the HH dataset is widely cited, used to train many models, and treated as a safety benchmark.

What carries the argument

The argument is carried by three interlocking pieces: a manual annotation taxonomy that labels chosen harmless responses as acceptable, unhelpful, or harmful; a controlled training comparison using DPO on the full HH dataset versus a filtered variant that removes every sample whose prompt contains any of 50 identity terms from Dixon et al. (2018); and evaluation on XS-ID, a new benchmark of 2,500 prompts that substitutes each of those 50 identity terms into XSTest templates to measure refusal rates on safe and unsafe queries. The identity-term filter is the key lever of the causal argument: the drop in refusal rates on safe identity prompts when these samples are removed is interpreted as evidence that identity terms are the mechanism linking the dataset to disparate safety behavior.

What would settle it

Train a model on the HH dataset with an equal number of non-identity red-teaming samples removed (matched for harm category and severity), and check whether the refusal rate on safe identity prompts still drops as much as it does for the identity-filtered variant; if the drop is the same, the identity-association mechanism is not the cause. A second test is to see whether refusal rates on safe identity prompts are actually predicted by the frequency of those identity terms in the harmless split; a poor fit would undermine the claimed correlation.

Watch

Extended reading notes

Core claim

The central claim is that conceptualization failures and quality issues in the HH dataset create additional harms by leading to disparate safety behaviors across demographic groups. The paper reports that only 11.8% of the chosen harmless responses are acceptable, 44.5% are unhelpful, and 43.7% are harmful, showing that relative preferences often amounted to choosing between two harmful or failing responses. In experiments, models trained with DPO on the full HH dataset refuse safe identity-related prompts (for example, queries about African American or Hispanic groups) at substantially higher rates than models trained on a version filtered to remove all samples containing any of 50 identity terms, and the pattern of refusal rates correlates with how often those terms appear in the harmless split. The authors interpret this as evidence that the dataset encodes spurious associations between identity terms and harmfulness, and that the "safety bundle" framing that lumps all harms together without severity or context contributes to the problem.

Load-bearing premise

The causal claim rests on the assumption that removing all samples containing identity terms isolates the effect of identity-harm associations, rather than merely reducing the overall amount of safety training and red-teaming content about racism, hate speech, and other harm categories.

Editorial extensions

If this is right

Models trained on the full HH dataset learn to refuse safe queries much more often than models trained only on the helpful portion, and the word "sorry" appears in about 20% of GPT-Full outputs versus 0.1% of GPT-Help outputs.
Removing identity-term samples lowers refusal rates on safe identity prompts for almost all of the 50 identity terms tested, indicating a spurious identity-harm association in the data.
The "safety bundle" that gives equal weight to trivial and severe harms makes the dataset unreliable as a safety benchmark, since samples are not differentiated by severity or context.
The helpfulness-harmlessness trade-off is not an inevitable consequence of safety training; it is amplified by low-quality relative preferences, so reframing safety as context-specific quality could reduce the trade-off.
A survey of 100 influential citing papers shows that fewer than half use the dataset for safety work and half treat the trade-off as unavoidable, suggesting the dataset's limitations have been inherited by the literature.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A cleaner causal test would control for semantic context rather than just term presence, for example by filtering red-teaming prompts that mention identity terms in racist language and comparing refusal rates with a matched non-identity control set; this distinction is not made in the paper.
The same audit method could detect hidden demographic associations in other preference datasets built on relative harm preferences, such as datasets labeled only as "less harmful than the alternative".
If the association generalizes, newer instruction-tuned models that undergo additional alignment stages after HH training may still show residual refusal disparities, which would suggest the effect is not erased by later fine-tuning.
A practical extension would be to measure refusal rates separately for each identity term and each harm category in the safe prompts, to identify which specific terms drive the disparity and whether the pattern is stable across base models.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The headline: this is the first serious external audit of the HH preference dataset, and the content-analysis part is worth taking seriously. The causal claim in the abstract—that dataset quality issues 'create additional harms by leading to disparate safety behaviors across demographic groups'—is not backed by the experiments as designed. Read it for the descriptive findings, not for the causal conclusion.

What's new: they annotate 4.2k chosen 'harmless' responses and find only ~12% acceptable, ~44% unhelpful, ~44% harmful. That's a striking result about a dataset cited over a thousand times and used to train many models. The taxonomy (acceptable/unhelpful/harmful) is simple but useful. The XS-ID suite extends XSTest to 50 identity terms, which is a reasonable evaluation tool. The citation survey of 100 papers is a nice addition, though it is basically a categorization exercise and does not deeply analyze how the trade-off framing propagates.

What's soft: the Full-vs-Filtered comparison in Section 4.4 cannot isolate identity-term associations. Filtering out all prompts containing any of 50 identity terms also removes a large block of red-teaming content about racism, hate speech, and so on, and shrinks the training set. The refusal-rate drops could come from reduced safety training generally, not from removing identity-harm co-occurrences. Pythia even reverses direction (16.00% to 17.17%), which the paper mentions only as 'does not change much.' No confidence intervals or significance tests are reported anywhere, and refusal rates are based on 25 generations per prompt, so they may be noisy. The annotation reliability is moderate (Cohen's kappa 0.50), and the BERT classifier for the 'acceptable' class gets 52% accuracy, so the automated full-dataset distribution is shaky. The manual numbers are more trustworthy, but even there, the annotators are the authors and each item was annotated once; the appendix acknowledges this but does not fully resolve it.

The paper's own Limitations section is honest about the single-annotation and author-annotator issues. Good to see that.

Bottom line: the descriptive audit is a genuine contribution and will be useful to anyone building or evaluating safety preference data. The causal story about disparate safety behaviors should be softened to a hypothesis. I would send this to review, but with a request that the authors either add a proper control (e.g., removing equal-size random samples or filtering non-identity safety content) or explicitly reframe Section 4 as demonstrating correlation, not causation.

Who it is for: people working on RLHF data quality, safety alignment, and dataset audits. It is worth bringing to a reading group as a case study in dataset auditing.

Recommendation: accept with major revision, conditional on re-analysis or reframing of the causal claim.

Referee Report

4 major / 4 minor

Summary. This paper presents a three-part audit of Anthropic's Helpful and Harmless (HH) dataset: (1) an exploratory content analysis and a manual taxonomy of harmless-preference conversations, reporting that only 11.8% of chosen harmless responses are judged acceptable, 44.5% unhelpful, and 43.7% harmful; (2) preference-optimization experiments comparing models trained on Helpful Only, HH Full, and HH Filtered variants, reporting refusal rates on XSTest and on a new XS-ID benchmark; and (3) a survey of 100 papers citing Bai et al. (2022). The central claim is that conceptualization failures and quality issues in the HH dataset can create additional harms by leading to disparate safety behaviors across demographic groups.

Significance. The paper is valuable as one of the first external audits of a preference dataset, and its descriptive findings about HH are important for the alignment community. The content audit is independently informative even if the causal experiment were weakened, and the community survey adds a useful perspective on how the dataset is cited and used. The authors ship their code, evaluate three model families, and include appendices with examples and additional results, which supports reproducibility. If the causal claim were established, the paper would provide a concrete mechanism connecting dataset quality to disparate safety harms; however, as it stands, the experimental design does not isolate that mechanism.

major comments (4)

[§4.4, Table 1, Figure 3] The Full-vs-Filtered comparison is confounded. HH Filtered removes every training sample whose prompt contains any of the 50 identity terms (§4.1), which simultaneously reduces dataset size from 160,800 to 154,076 samples (Table 3) and removes a large block of safety-relevant red-teaming content, including prompts about racism, hate speech, and other harm categories. The lower refusal rates for GPT-Filtered and OPT-Filtered are equally consistent with reduced safety training overall, not with removal of identity-harm associations specifically. The Pythia result even reverses direction (17.17% for Pythia-Filtered vs 16.00% for Pythia-Full), which the paper does not explain. To support the causal claim, the authors should add control conditions that hold dataset size and general safety content roughly fixed, for example a same-size random subset of HH Full and a filtered variant that removes a matched number of non-identity safety prompts.
[§4.3, §4.4] Refusal rates are reported without confidence intervals or significance tests. With 25 generations per prompt and refusal detection based on string matching, the differences in Table 1, such as GPT-Full 33.74% vs GPT-Filtered 23.18%, and the per-identity differences in Figure 3, may be within sampling noise. The paper should report bootstrap confidence intervals, standard errors, or a significance test over prompts, and should state the number of evaluation prompts behind each percentage. This is needed before interpreting the magnitudes as evidence of disparate safety behavior.
[§3.2.2, Appendix D, Appendix E] The headline content-quality rates rest on single-author annotations with a moderate Cohen's kappa of 0.50 and an automated classifier that reaches only 52% accuracy on the 'acceptable' class. The paper's own Limitations section acknowledges single annotation. Because the taxonomy is defined by the authors and the manual sample is a single pass, the exact values of 11.8%, 44.5%, and 43.7% should be presented with uncertainty intervals or at least explicitly framed as point estimates from a moderate-reliability coding scheme. The current wording in §3.3 treats these as firm estimates, which overstates their precision.
[Abstract, §6, §4.4] The causal language in the abstract and discussion is stronger than the evidence. The experiments show that filtering identity-term prompts changes refusal rates in some models, but the abstract claims that quality issues 'can create additional harms by leading to disparate safety behaviors across demographic groups.' Given the confound in §4.4 and the lack of significance testing, the appropriate conclusion is that the HH dataset is associated with elevated refusal rates on identity-related safe prompts in some models. The causal claim should be softened or supported by the additional control experiments described above.

minor comments (4)

[Appendix G, Table 3] The caption contains a duplicated article: 'the the Helpful and Harmless dataset' should be 'the Helpful and Harmless dataset.'
[§3.2.2, Appendix D] The annotation protocol is described first as a two-stage process with 400 conversations to create a taxonomy and then 4.2k annotations, but Appendix D says 'randomly annotating 400 data points to examine potential disagreements.' Please clarify whether the 400 disagreement samples are separate from the 4.2k main sample and how they were selected.
[§3.2.1, Figure 2] The 'scale of harms' from 'irrelevant' to 'violent' in Figure 2 is presented without a definition of how severity is ordered. Please specify the criteria used to order the clusters along this scale, since some categories such as 'sexual content' and 'privacy' are not inherently on the same continuum as physical violence.
[§5, Appendix J] The survey of the 100 most influential papers relies on Google Scholar's relevance ranking, but the exact query, date of retrieval, and inclusion criteria are not reported. Adding these details would make the selection reproducible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's content audit and training ablations are empirical and not derived from their own inputs.

full rationale

This paper is an empirical audit rather than a derivation. The content-quality findings in Section 3.2.2 come from manual annotation of a random sample and from a classifier whose accuracy is reported separately; the percentages are measurements, not fitted parameters renamed as predictions. The training experiments in Section 4 use externally defined XSTest and RealToxicityPrompts benchmarks, and the XS-ID extension is a templatic transformation of XSTest. The HH Filtered condition is an ablation that removes identity-term prompts, and although the filter and XS-ID share the same 50 identity terms, the measured refusal rates are empirical outcomes rather than consequences forced by construction. In fact, the paper itself notes that Pythia-Filtered does not show the expected reduction, which confirms the result is not tautological. The Full-versus-Filtered comparison has a plausible confounding alternative explanation (the filter also removes general red-teaming content and shrinks the dataset), but this is a validity or correctness concern, not circularity. The only self-citation, Chehbouni et al. (2024), is used to connect the results to prior documentation of disparate safety behaviors, but the present experiments stand independently and do not rely on that citation for their evidence. No step in the paper derives its conclusion from its own definition or from a self-citation chain.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a small manually annotated sample (4.2k conversations, single annotator, kappa 0.50), a confounded filtered-dataset comparison, and point estimates without confidence intervals. No mathematical derivation or fitted physical parameters are involved; the main fitted artifact is a classifier used to extend the manual labels. No new theoretical entities are introduced.

free parameters (3)

DPO beta, batch size, learning rate = beta=0.1, batch_size=64, learning_rate=1e-6
Adopted from Rafailov et al. (2024) after a two-point hyperparameter search on GPT-2 only; used for all models. The central results may depend on this choice, but the search was not tuned to the safety evaluation outcomes.
Number of K-means clusters for red-teaming prompts = 10
Chosen by the authors to group red-teaming prompts into interpretable topics in Section 3.2.1; different k would change the topic labels but not the core quality findings.
BERT classifier for automatic harmless-category annotation = accuracy: harmful 81%, unhelpful 79%, acceptable 52%
Trained on the authors' manual labels and used to extend the distribution to the full dataset and to other preference datasets in Appendix E. The low acceptable-class accuracy limits the automated distribution estimates.

assumptions (4)

domain assumption String matching on refusal phrases is a valid operationalization of safety and helpfulness.
Used in Section 4.3 to label model outputs as refusal or compliance; misclassifications would affect all reported refusal rates.
ad hoc to paper The removed identity-term prompts in HH Filtered isolate the mechanism by which models associate identity terms with harmfulness.
Section 4.1 removes every sample whose prompt contains any of 50 identity terms; Section 4.4 attributes the refusal-rate drop to identity associations, but the filter also removes safety-relevant red-teaming content generally.
domain assumption The three-category manual taxonomy (acceptable, unhelpful, harmful) is reliable enough to estimate dataset composition.
Section 3.2.2 and Appendix D: single-annotation protocol with Cohen's kappa 0.50; the authors argue the stable distribution supports reliability.
domain assumption The base models (GPT-2 large, Pythia 2.8B, OPT 2.7B) had no prior safety training, so learned safety behaviors are attributable to the HH dataset.
Section 4.2: older models selected for this reason; residual base-model biases remain possible, especially for identity-related prompts.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Beyond the Safety Bundle: Auditing the Helpful and Harmless Dataset." pith.science (2026). https://pith.science/paper/SAR3ZRQV

@misc{pith2026241108243,
  author       = {Pith},
  title        = {Pith review of: Beyond the Safety Bundle: Auditing the Helpful and Harmless Dataset},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SAR3ZRQV}},
  note         = {Machine review of arXiv:2411.08243}
}

read the original abstract

In an effort to mitigate the harms of large language models (LLMs), learning from human feedback (LHF) has been used to steer LLMs towards outputs that are intended to be both less harmful and more helpful. Despite the widespread adoption of LHF in practice, the quality of this feedback and its effectiveness as a safety mitigation technique remain unclear. This study addresses these issues by auditing the widely-used Helpful and Harmless (HH) dataset by Anthropic. Our work includes: (1) a thorough investigation of the dataset's content through both manual and automated evaluation; (2) experiments demonstrating the dataset's impact on models' safety; and (3) an analysis of the 100 most influential papers citing this dataset. Through our audit, we showcase how conceptualization failures and quality issues identified in the HH dataset can create additional harms by leading to disparate safety behaviors across demographic groups. Our findings highlight the need for more nuanced, context-sensitive approaches to safety mitigation in LLMs.

Figures

Figures reproduced from arXiv: 2411.08243 by the authors.

**Figure 2.** Example of different red-teaming queries. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** We compare the refusal rates to respond to [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (10 more)

**Figure 4.** Figure 4: Most Frequent Words Comparatively in the [PITH_FULL_IMAGE:figures/full_fig_p022_4.png]

**Figure 5.** Figure 5: Most common topics in the harmless dataset, along with the most important words in each topic. tion. This also underscores the subjective nature of harm classification and highlights the challenges of evaluation in this context. For instance, one annotator classified …

**Figure 6.** Figure 6: Most common topics in the helpful dataset, along with the most important words in each topic. especially considering the accuracy results of the trained BERT models on the acceptable category. Furthermore, we use this newly trained classifier to evaluate other preferen…

**Figure 7.** Figure 7: We perform a principal component analysis (PCA) of our clusters and plot the results. Each point [PITH_FULL_IMAGE:figures/full_fig_p026_7.png]

**Figure 8.** Figure 8: Pie Chart of the Distribution of our labels [PITH_FULL_IMAGE:figures/full_fig_p027_8.png]

**Figure 9.** Figure 9: Pie Chart of the Distribution of predicted [PITH_FULL_IMAGE:figures/full_fig_p027_9.png]

**Figure 10.** Figure 10: Conversation extracted from the Harmless dataset. In green the option identified as harmless. [PITH_FULL_IMAGE:figures/full_fig_p028_10.png]

**Figure 11.** Figure 11: Percentage of samples in the harmless and helpful splits of the HH dataset whose prompts contain certain [PITH_FULL_IMAGE:figures/full_fig_p030_11.png]

**Figure 12.** Figure 12: We compare the refusal rates to respond to safe prompts of a model trained on the full HH dataset to a [PITH_FULL_IMAGE:figures/full_fig_p030_12.png]

**Figure 13.** Figure 13: This word cloud illustrates the most frequent terms found within the abstracts of the papers included in [PITH_FULL_IMAGE:figures/full_fig_p031_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 1 canonical work pages

[2]

Transactions of the Association for Computational Linguistics, 11:1643–1668

Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation. Transactions of the Association for Computational Linguistics, 11:1643–1668. Luke Friedman, Sameer Ahuja, David Allen, Zhen- ning Tan, Hakim Sidahmed, Changbo Long, Jun Xie, Gabriel Schubiner, Ajay Patel, Harsh Lara, Brian Chu, Zexi Chen, and Manoj Tiwari. 2023. Le...

arXiv 2023
[4]

InProceedings of the 16th ACM Workshop on Artificial Intelligence and Secu- rity, AISec ’23, page 79–90, New York, NY , USA

Not what you’ve signed up for: Compromis- ing real-world llm-integrated applications with indi- rect prompt injection. InProceedings of the 16th ACM Workshop on Artificial Intelligence and Secu- rity, AISec ’23, page 79–90, New York, NY , USA. Association for Computing Machinery. Maarten Grootendorst. 2022. Bertopic: Neural topic modeling with a class-bas...

arXiv 2022
[5]

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi

Evaluating large language models on a highly- specialized topic, radiation oncology physics.Fron- tiers in Oncology, 13. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration.Preprint, arXiv:1904.09751. Shengran Hu and Jeff Clune. 2023. Thought cloning: Learning to think while acting by imitating...

arXiv 2020
[7]

Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz

The alignment problem from a deep learning perspective.Preprint, arXiv:2209.00626. Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023. Capabilities of gpt-4 on medical challenge problems.Preprint, arXiv:2303.13375. Jesutofunmi A Omiye, Jenna C Lester, Simon Spichak, Veronica Rotemberg, and Roxana Daneshjou

arXiv 2023
[8]

Large language models propagate race-based medicine.NPJ Digital Medicine, 6(1):195. OpenAI. 2023. Gpt-4 technical report.Preprint, arXiv:2303.08774. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie S...

arXiv 2023
[9]

Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord

Direct preference optimization: Your language model is secretly a reward model.Advances in Neu- ral Information Processing Systems, 36. Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. 2023. Rewarded soups: towards pareto-optimal alignment by inter- polating weights fine-tuned on ...

arXiv 2023
[10]

Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, et al

Character-llm: A trainable agent for role- playing.Preprint, arXiv:2310.10158. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, et al. 2023. Towards understand- ing sycophancy in language models.arXiv preprint arXiv:2310.13548. Renee Shelby, Shalaleh R...

arXiv
[11]

Jailbroken: How does LLM safety training fail? InThirty-seventh Conference on Neural Infor- mation Processing Systems. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Can...

arXiv 2020

Show all 13 references

[12]

gayest" as a descriptor that means

Gptfuzzer: Red teaming large language mod- els with auto-generated jailbreak prompts.Preprint, arXiv:2309.10253. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024a. Self-rewarding language models. Preprint, arXiv:2...

2023 arXiv
[2020]

In total, the dataset contains 22K toxic prompts

contains 100K sentences extracted from the OpenWebText Corpus (Gokaslan and Cohen, 2019) – a dataset created by scrapping the content of out- bound hyperlinks found on Reddit – and their cor- responding toxicity score extracted using Perspec- tive API *. In total, the dataset ...

2022
[2022]

Association for Computational Linguistics

Are large pre-trained language models leaking your personal information? InFindings of the Asso- ciation for Computational Linguistics: EMNLP 2022, pages 2038–2047, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. Yangsibo Huang, Samyak Gupta, Mengzh...

2022 arXiv
[2023]

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer

Safe rlhf: Safe reinforcement learning from human feedback.Preprint, arXiv:2310.12773. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. InAdvances in Neural Information Processing Systems, volume 36, pages...

2023 arXiv
[2024]

Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz

Critic: Large language models can self- correct with tool-interactive critiquing.Preprint, arXiv:2305.11738. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz

arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [2]

Transactions of the Association for Computational Linguistics, 11:1643–1668

Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation. Transactions of the Association for Computational Linguistics, 11:1643–1668. Luke Friedman, Sameer Ahuja, David Allen, Zhen- ning Tan, Hakim Sidahmed, Changbo Long, Jun Xie, Gabriel Schubiner, Ajay Patel, Harsh Lara, Brian Chu, Zexi Chen, and Manoj Tiwari. 2023. Le...

arXiv 2023

[2] [4]

InProceedings of the 16th ACM Workshop on Artificial Intelligence and Secu- rity, AISec ’23, page 79–90, New York, NY , USA

Not what you’ve signed up for: Compromis- ing real-world llm-integrated applications with indi- rect prompt injection. InProceedings of the 16th ACM Workshop on Artificial Intelligence and Secu- rity, AISec ’23, page 79–90, New York, NY , USA. Association for Computing Machinery. Maarten Grootendorst. 2022. Bertopic: Neural topic modeling with a class-bas...

arXiv 2022

[3] [5]

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi

Evaluating large language models on a highly- specialized topic, radiation oncology physics.Fron- tiers in Oncology, 13. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration.Preprint, arXiv:1904.09751. Shengran Hu and Jeff Clune. 2023. Thought cloning: Learning to think while acting by imitating...

arXiv 2020

[4] [7]

Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz

The alignment problem from a deep learning perspective.Preprint, arXiv:2209.00626. Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023. Capabilities of gpt-4 on medical challenge problems.Preprint, arXiv:2303.13375. Jesutofunmi A Omiye, Jenna C Lester, Simon Spichak, Veronica Rotemberg, and Roxana Daneshjou

arXiv 2023

[5] [8]

Large language models propagate race-based medicine.NPJ Digital Medicine, 6(1):195. OpenAI. 2023. Gpt-4 technical report.Preprint, arXiv:2303.08774. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie S...

arXiv 2023

[6] [9]

Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord

Direct preference optimization: Your language model is secretly a reward model.Advances in Neu- ral Information Processing Systems, 36. Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. 2023. Rewarded soups: towards pareto-optimal alignment by inter- polating weights fine-tuned on ...

arXiv 2023

[7] [10]

Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, et al

Character-llm: A trainable agent for role- playing.Preprint, arXiv:2310.10158. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, et al. 2023. Towards understand- ing sycophancy in language models.arXiv preprint arXiv:2310.13548. Renee Shelby, Shalaleh R...

arXiv

[8] [11]

Jailbroken: How does LLM safety training fail? InThirty-seventh Conference on Neural Infor- mation Processing Systems. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Can...

arXiv 2020