{"id":"b174a52d-235d-4ca1-9a9f-3060e55df553","arxiv_id":"2411.08243","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Most chosen 'harmless' replies in the HH dataset are unhelpful or harmful, and training on it causes unequal, exaggerated refusal rates across demographic groups.","lead":"A widely used AI safety training dataset, Anthropic's Helpful and Harmless dataset, turns out to contain many unhelpful or even harmful example answers. Auditing it shows that models trained on it refuse safe questions about certain demographic groups much more often, which can create new harms.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The §4.4 Full-vs-Filtered comparison cannot isolate identity-term associations: filtering removes general red-teaming content and shrinks the dataset, so the refusal drop may reflect reduced safety training rather than demographically disparate harm associations.","rationale":"I agree with the reader's weakest assumption: the §4.4 comparison is the load-bearing link between the audit findings and the headline claim. The content findings—11.8% acceptable, 44.5% unhelpful, 43.7% harmful—are striking and, as the reader notes, likely robust in direction, though limited by single-annotation and moderate kappa. The community survey (§5) is descriptive and does not carry the causal weight. The causal claim about demographic disparities, however, depends entirely on the Filtered variant isolating identity-term associations. It does not: the filter is a coarse deletion of all identity-mentioning samples, which also deletes safety-relevant red-teaming content and changes dataset size. The paper's Limitations section (§8) acknowledges the small, single-annotator sample but does not address this identification problem. The proposed control experiment would settle whether the refusal drop is identity-specific or a general consequence of reducing safety training. Since the content audit remains valuable, a conditional verdict is appropriate; the causal claim should be softened or re-analyzed before acceptance, which matches the reader's recommendation.","tokens_in":37686,"tokens_out":4118,"duration_ms":41283,"concrete_test":"Train a fourth model, HH Filtered-Other, by removing from HH Full the same number of samples as HH Filtered but selecting red-teaming samples whose prompts do not contain any of the 50 identity terms and are matched on harm cluster (from §3.2.1), prompt length, and dataset split. Evaluate all models on the XS-ID safe prompts with at least 25 generations per prompt and bootstrap confidence intervals. If HH Filtered-Other produces refusal rates on identity-term prompts as low as HH Filtered, the Full-vs-Filtered drop is a generic safety-training-volume effect, invalidating the identity-association interpretation. If, instead, only HH Filtered shows the drop, the identity-specific mechanism is supported. A secondary check: also compute refusal rates on matched non-identity safe prompts to verify the effect is specific to identity terms.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central causal claim—that quality issues in HH 'create additional harms by leading to disparate safety behaviors across demographic groups'—rests on the comparison in §4.4 between models trained on HH Full and HH Filtered (Table 1, Figure 3). HH Filtered removes every sample whose prompt contains any of 50 identity terms (§4.1). This manipulation changes three things at once: (1) it removes identity-harm co-occurrences, (2) it removes a large block of safety-relevant red-teaming content that happens to co-occur with identity terms (e.g., prompts about racism, hate speech, and discrimination), and (3) it reduces training set size from 160,800 to 154,076 samples (Table 3). The observed pattern—lower refusal to safe prompts in GPT-Filtered (23.18% vs 33.74%) and OPT-Filtered (21.41% vs 25.95%)—is equally consistent with any of these three changes. For Pythia the direction reverses (17.17% vs 16.00%), which the paper does not explain. No control condition holds dataset size or general safety content fixed, and no confidence intervals or significance tests are reported, so the differential refusal rates across identity terms in Figure 3 cannot be causally attributed to spurious identity-harm associations. Without this causal link, the headline claim of disparate safety behaviors is not supported by the experiments, although the content audit findings (§3.2.2) remain independent evidence of dataset quality problems.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a three-part audit of Anthropic's Helpful and Harmless (HH) dataset: (1) an exploratory content analysis and a manual taxonomy of harmless-preference conversations, reporting that only 11.8% of chosen harmless responses are judged acceptable, 44.5% unhelpful, and 43.7% harmful; (2) preference-optimization experiments comparing models trained on Helpful Only, HH Full, and HH Filtered variants, reporting refusal rates on XSTest and on a new XS-ID benchmark; and (3) a survey of 100 papers citing Bai et al. (2022). The central claim is that conceptualization failures and quality issues in the HH dataset can create additional harms by leading to disparate safety behaviors across demographic groups.","tokens_in":37980,"tokens_out":4137,"duration_ms":41364,"significance":"The paper is valuable as one of the first external audits of a preference dataset, and its descriptive findings about HH are important for the alignment community. The content audit is independently informative even if the causal experiment were weakened, and the community survey adds a useful perspective on how the dataset is cited and used. The authors ship their code, evaluate three model families, and include appendices with examples and additional results, which supports reproducibility. If the causal claim were established, the paper would provide a concrete mechanism connecting dataset quality to disparate safety harms; however, as it stands, the experimental design does not isolate that mechanism.","major_comments":[{"comment":"The Full-vs-Filtered comparison is confounded. HH Filtered removes every training sample whose prompt contains any of the 50 identity terms (§4.1), which simultaneously reduces dataset size from 160,800 to 154,076 samples (Table 3) and removes a large block of safety-relevant red-teaming content, including prompts about racism, hate speech, and other harm categories. The lower refusal rates for GPT-Filtered and OPT-Filtered are equally consistent with reduced safety training overall, not with removal of identity-harm associations specifically. The Pythia result even reverses direction (17.17% for Pythia-Filtered vs 16.00% for Pythia-Full), which the paper does not explain. To support the causal claim, the authors should add control conditions that hold dataset size and general safety content roughly fixed, for example a same-size random subset of HH Full and a filtered variant that removes a matched number of non-identity safety prompts.","section":"§4.4, Table 1, Figure 3"},{"comment":"Refusal rates are reported without confidence intervals or significance tests. With 25 generations per prompt and refusal detection based on string matching, the differences in Table 1, such as GPT-Full 33.74% vs GPT-Filtered 23.18%, and the per-identity differences in Figure 3, may be within sampling noise. The paper should report bootstrap confidence intervals, standard errors, or a significance test over prompts, and should state the number of evaluation prompts behind each percentage. This is needed before interpreting the magnitudes as evidence of disparate safety behavior.","section":"§4.3, §4.4"},{"comment":"The headline content-quality rates rest on single-author annotations with a moderate Cohen's kappa of 0.50 and an automated classifier that reaches only 52% accuracy on the 'acceptable' class. The paper's own Limitations section acknowledges single annotation. Because the taxonomy is defined by the authors and the manual sample is a single pass, the exact values of 11.8%, 44.5%, and 43.7% should be presented with uncertainty intervals or at least explicitly framed as point estimates from a moderate-reliability coding scheme. The current wording in §3.3 treats these as firm estimates, which overstates their precision.","section":"§3.2.2, Appendix D, Appendix E"},{"comment":"The causal language in the abstract and discussion is stronger than the evidence. The experiments show that filtering identity-term prompts changes refusal rates in some models, but the abstract claims that quality issues 'can create additional harms by leading to disparate safety behaviors across demographic groups.' Given the confound in §4.4 and the lack of significance testing, the appropriate conclusion is that the HH dataset is associated with elevated refusal rates on identity-related safe prompts in some models. The causal claim should be softened or supported by the additional control experiments described above.","section":"Abstract, §6, §4.4"}],"minor_comments":[{"comment":"The caption contains a duplicated article: 'the the Helpful and Harmless dataset' should be 'the Helpful and Harmless dataset.'","section":"Appendix G, Table 3"},{"comment":"The annotation protocol is described first as a two-stage process with 400 conversations to create a taxonomy and then 4.2k annotations, but Appendix D says 'randomly annotating 400 data points to examine potential disagreements.' Please clarify whether the 400 disagreement samples are separate from the 4.2k main sample and how they were selected.","section":"§3.2.2, Appendix D"},{"comment":"The 'scale of harms' from 'irrelevant' to 'violent' in Figure 2 is presented without a definition of how severity is ordered. Please specify the criteria used to order the clusters along this scale, since some categories such as 'sexual content' and 'privacy' are not inherently on the same continuum as physical violence.","section":"§3.2.1, Figure 2"},{"comment":"The survey of the 100 most influential papers relies on Google Scholar's relevance ranking, but the exact query, date of retrieval, and inclusion criteria are not reported. Adding these details would make the selection reproducible.","section":"§5, Appendix J"}],"recommendation":"major_revision","confidential_remarks":"The descriptive audit and the survey are likely to be useful contributions, and the code release is a strength. In my view the paper needs a major revision primarily because the central causal claim is not supported by the current Full-vs-Filtered comparison. The content findings can stand on their own if the authors frame them as descriptive estimates rather than exact figures. I would not reject the paper, because the confound is addressable within the scope of the manuscript through additional control experiments and more cautious language."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline: this is the first serious external audit of the HH preference dataset, and the content-analysis part is worth taking seriously. The causal claim in the abstract—that dataset quality issues 'create additional harms by leading to disparate safety behaviors across demographic groups'—is not backed by the experiments as designed. Read it for the descriptive findings, not for the causal conclusion.\n\nWhat's new: they annotate 4.2k chosen 'harmless' responses and find only ~12% acceptable, ~44% unhelpful, ~44% harmful. That's a striking result about a dataset cited over a thousand times and used to train many models. The taxonomy (acceptable/unhelpful/harmful) is simple but useful. The XS-ID suite extends XSTest to 50 identity terms, which is a reasonable evaluation tool. The citation survey of 100 papers is a nice addition, though it is basically a categorization exercise and does not deeply analyze how the trade-off framing propagates.\n\nWhat's soft: the Full-vs-Filtered comparison in Section 4.4 cannot isolate identity-term associations. Filtering out all prompts containing any of 50 identity terms also removes a large block of red-teaming content about racism, hate speech, and so on, and shrinks the training set. The refusal-rate drops could come from reduced safety training generally, not from removing identity-harm co-occurrences. Pythia even reverses direction (16.00% to 17.17%), which the paper mentions only as 'does not change much.' No confidence intervals or significance tests are reported anywhere, and refusal rates are based on 25 generations per prompt, so they may be noisy. The annotation reliability is moderate (Cohen's kappa 0.50), and the BERT classifier for the 'acceptable' class gets 52% accuracy, so the automated full-dataset distribution is shaky. The manual numbers are more trustworthy, but even there, the annotators are the authors and each item was annotated once; the appendix acknowledges this but does not fully resolve it.\n\nThe paper's own Limitations section is honest about the single-annotation and author-annotator issues. Good to see that.\n\nBottom line: the descriptive audit is a genuine contribution and will be useful to anyone building or evaluating safety preference data. The causal story about disparate safety behaviors should be softened to a hypothesis. I would send this to review, but with a request that the authors either add a proper control (e.g., removing equal-size random samples or filtering non-identity safety content) or explicitly reframe Section 4 as demonstrating correlation, not causation.\n\nWho it is for: people working on RLHF data quality, safety alignment, and dataset audits. It is worth bringing to a reading group as a case study in dataset auditing.\n\nRecommendation: accept with major revision, conditional on re-analysis or reframing of the causal claim.","headline":"First real audit of the HH preference dataset, with striking descriptive findings; the causal claim about disparate safety behaviors rests on a confounded experiment and should be softened.","tokens_in":38536,"tokens_out":1876,"would_cite":true,"duration_ms":18001,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Auditing the Helpful and Harmless dataset shows that its \"harmless\" responses are mostly low quality, and models trained on it learn to refuse safe prompts about demographic groups more often, creating disparate safety behavior.","keywords":["dataset audit","helpful harmless dataset","human preference data","safety alignment","exaggerated safety behavior","demographic disparity","direct preference optimization","refusal behavior"],"falsifier":"Train a model on the HH dataset with an equal number of non-identity red-teaming samples removed (matched for harm category and severity), and check whether the refusal rate on safe identity prompts still drops as much as it does for the identity-filtered variant; if the drop is the same, the identity-association mechanism is not the cause. A second test is to see whether refusal rates on safe identity prompts are actually predicted by the frequency of those identity terms in the harmless split; a poor fit would undermine the claimed correlation.","tokens_in":37467,"feed_emoji":"🛡️","tokens_out":9295,"duration_ms":77639,"temperature":0.7,"pith_summary":"This paper audits the Helpful and Harmless (HH) dataset, a widely used human-preference dataset for steering language models toward safe behavior. It finds that the dataset's chosen \"harmless\" responses are rarely acceptable: only about 12 percent truly refuse harm, while roughly 44 percent are unhelpful failures and 44 percent are themselves harmful. Training three base models with direct preference optimization on the full dataset makes them refuse safe prompts that mention demographic identity terms at far higher rates than models trained without identity-containing samples, and these refusal rates track the overrepresentation of identity terms in the harmless split. The paper argues that the dataset thereby teaches models to associate demographic terms with harmfulness, producing disparate safety behavior across groups. This matters because the HH dataset is widely cited, used to train many models, and treated as a safety benchmark.","feed_headline":"Audit: safety dataset teaches models to refuse benign identity queries","feed_subtitle":"Only 12% of \"harmless\" responses are truly acceptable, and models over-refuse safe demographic queries.","key_machinery":"The argument is carried by three interlocking pieces: a manual annotation taxonomy that labels chosen harmless responses as acceptable, unhelpful, or harmful; a controlled training comparison using DPO on the full HH dataset versus a filtered variant that removes every sample whose prompt contains any of 50 identity terms from Dixon et al. (2018); and evaluation on XS-ID, a new benchmark of 2,500 prompts that substitutes each of those 50 identity terms into XSTest templates to measure refusal rates on safe and unsafe queries. The identity-term filter is the key lever of the causal argument: the drop in refusal rates on safe identity prompts when these samples are removed is interpreted as evidence that identity terms are the mechanism linking the dataset to disparate safety behavior.","core_discovery":"The central claim is that conceptualization failures and quality issues in the HH dataset create additional harms by leading to disparate safety behaviors across demographic groups. The paper reports that only 11.8% of the chosen harmless responses are acceptable, 44.5% are unhelpful, and 43.7% are harmful, showing that relative preferences often amounted to choosing between two harmful or failing responses. In experiments, models trained with DPO on the full HH dataset refuse safe identity-related prompts (for example, queries about African American or Hispanic groups) at substantially higher rates than models trained on a version filtered to remove all samples containing any of 50 identity terms, and the pattern of refusal rates correlates with how often those terms appear in the harmless split. The authors interpret this as evidence that the dataset encodes spurious associations between identity terms and harmfulness, and that the \"safety bundle\" framing that lumps all harms together without severity or context contributes to the problem.","pith_inferences":["A cleaner causal test would control for semantic context rather than just term presence, for example by filtering red-teaming prompts that mention identity terms in racist language and comparing refusal rates with a matched non-identity control set; this distinction is not made in the paper.","The same audit method could detect hidden demographic associations in other preference datasets built on relative harm preferences, such as datasets labeled only as \"less harmful than the alternative\".","If the association generalizes, newer instruction-tuned models that undergo additional alignment stages after HH training may still show residual refusal disparities, which would suggest the effect is not erased by later fine-tuning.","A practical extension would be to measure refusal rates separately for each identity term and each harm category in the safe prompts, to identify which specific terms drive the disparity and whether the pattern is stable across base models."],"forward_implications":["Models trained on the full HH dataset learn to refuse safe queries much more often than models trained only on the helpful portion, and the word \"sorry\" appears in about 20% of GPT-Full outputs versus 0.1% of GPT-Help outputs.","Removing identity-term samples lowers refusal rates on safe identity prompts for almost all of the 50 identity terms tested, indicating a spurious identity-harm association in the data.","The \"safety bundle\" that gives equal weight to trivial and severe harms makes the dataset unreliable as a safety benchmark, since samples are not differentiated by severity or context.","The helpfulness-harmlessness trade-off is not an inevitable consequence of safety training; it is amplified by low-quality relative preferences, so reframing safety as context-specific quality could reduce the trade-off.","A survey of 100 influential citing papers shows that fewer than half use the dataset for safety work and half treat the trade-off as unavoidable, suggesting the dataset's limitations have been inherited by the literature."],"supporting_citations":[{"why":"Provides the Helpful and Harmless dataset itself, the primary object of the audit and the training data for all experiments.","marker":"Bai et al., 2022"},{"why":"Supplies XSTest, the benchmark for exaggerated safety behaviors, and the template prompts from which XS-ID is constructed.","marker":"Röttger et al., 2023"},{"why":"Documents disparate exaggerated safety behaviors for marginalized communities that this paper aims to trace back to dataset content.","marker":"Chehbouni et al., 2024"},{"why":"Provides the 50 identity terms used to build the filtered training variant and the XS-ID evaluation prompts.","marker":"Dixon et al., 2018"},{"why":"Supplies the DPO training procedure, hyperparameters, and reference implementation used in the safety experiments.","marker":"Rafailov et al., 2024"},{"why":"Provides RealToxicityPrompts, used as a downstream toxicity evaluation of the trained models.","marker":"Gehman et al., 2020"}],"fun_headline_variants":["Safety dataset audit: only 12% of 'harmless' responses acceptable","HH dataset: quality flaws cause models to over-refuse identity queries","Audit: safety bundle leads to disparate refusals across demographic groups","Only 12% of 'harmless' responses acceptable; models over-refuse identity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The causal claim rests on the assumption that removing all samples containing identity terms isolates the effect of identity-harm associations, rather than merely reducing the overall amount of safety training and red-teaming content about racism, hate speech, and other harm categories.","fun_headline_variants_meta":{"raw":{"variants":["Safety dataset audit: only 12% of 'harmless' responses acceptable","HH dataset: quality flaws cause models to over-refuse identity queries","Audit: safety bundle leads to disparate refusals across demographic groups","Only 12% of 'harmless' responses acceptable; models over-refuse identity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000655,"raw_usage":{"total_tokens":2980,"prompt_tokens":904,"completion_tokens":2076,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":1994}},"tokens_in":520,"tokens_out":2076,"duration_ms":15185,"temperature":1.0,"reasoning_tokens":1994,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:48:29.862882+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a model on the HH dataset with an equal number of non-identity red-teaming samples removed (matched for harm category and severity), and check whether the refusal rate on safe identity prompts still drops as much as it does for the identity-filtered variant; if the drop is the same, the identity-association mechanism is not the cause. A second test is to see whether refusal rates on safe identity prompts are actually predicted by the frequency of those identity terms in the harmless split; a poor fit would undermine the claimed correlation.","supporting_citations":[],"review_version":1}