Pith. sign in

REVIEW 4 major objections 6 minor 44 references

Towards Effective Discrimination Testing for Generative AI

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that four widely used GenAI bias tests—ROUGE equality, red teaming, single-turn evaluation, and default-parameter testing—can all rate a system as fair even when it produces discriminatory downstream outcomes, and that…

desk verdict A timely, honest critique of GenAI fairness testing with four illustrative case studies; the central hiring example is compelling but its quantitative gap is not yet evidence about real deployments. read the letter →

arxiv 2412.21052 v1 pith:GUL34QJF submitted 2024-12-30 cs.LG cs.AIcs.CY

Thomas P. Zollo , Nikita Rajaneesh , Richard Zemel , Talia B. Gillis , Emily Black This is my paper · ORCID
classification cs.LGcs.AIcs.CY
keywords generativeAIfairnesstestingdiscriminationdisparateimpactROUGEredteamingmulti-turnevaluationuser-modifiableparameters
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the standard ways of testing generative AI systems for bias are out of step with what anti-discrimination law actually cares about, so a model can pass a pre-deployment fairness review and still discriminate in real use. The authors identify four specific failure modes: equalizing a summarization metric like ROUGE across demographic groups does not equalize downstream hiring decisions; red-teaming results flip depending on which model generates the attacks; single-turn evaluations do not predict multi-turn behavior; and a user-adjustable parameter like the guidance scale in a diffusion model can suddenly make representational harms much worse for one group. If the argument holds, deployers and regulators who rely on these popular tests have no reliable way to tell a fair GenAI system from a discriminatory one. The paper closes with recommendations to build context-specific, robust testing frameworks that mirror deployment conditions.

What carries the argument

The paper's central device is a set of four case studies that pair a popular bias-measurement technique with a realistic downstream decision (hiring interviews, user-facing chatbot responses, multi-turn conversations, image generation). Each case study isolates the gap between the measured quantity and the harm the law cares about: ROUGE equality versus interview selection disparity; red-team attack success rate versus a stable fairness ranking; single-turn toxicity versus multi-turn toxicity; fixed hyperparameter testing versus user-modifiable generation settings. The fourth case also imports the notion of 'discrimination hacking' (d-hacking) from prior work, where practitioners intentionally or unintentionally choose a testing scheme that makes a biased model look fair; the paper's contribution is to show that generative AI's output flexibility and interaction modes enlarge the space of d-hacking. The load-bearing identity across all four studies is that the metric used for pre-deployment testing is not the quantity that determines whether an individual or group suffers an adverse allocative or representational outcome.

What would settle it

A direct human-subject test of the resume case would settle the main demonstration: give the same set of Llama-2-7B summaries, with and without names, to professional recruiters and measure interview-selection rates by group; if no systematic gap appears across groups when summaries are produced with names, the claim that the summarizer causes discriminatory downstream outcomes is not supported. For the red-teaming claim, a falsifier would be a randomized battery of many attack generators showing that the rank ordering of candidate models is stable across them, contradicting the near-arbitrary rankings reported for women.

Watch

Extended reading notes

Core claim

The paper's central claim is that existing bias assessment methods for generative AI are misaligned with the goals of anti-discrimination regulation, and that this misalignment is concrete enough to demonstrate in four controlled experiments. In the first, five candidate summarization models are compared by ROUGE score across racial groups; although all look fair on that metric, the model with the best ROUGE (Llama-2-7B) produces summaries that lead an LLM-simulated hiring manager to interview white candidates about 5% more often than Black or Hispanic candidates with identical resumes. In the second, the attack success rate of red-teaming tests for sexist output depends so strongly on which red-team model writes the prompts that one candidate (Llama3-8B) ranks as the least fair under some choices and the most fair under others. In the third, fairness rankings reverse when the same red-team attack is embedded in a multi-turn educational or medical conversation instead of a single turn. In the fourth, StableDiffusion's guidance scale, a parameter users can freely adjust, causes NSFW scores for Latina women to jump from near parity at scale 3.0 to dramatically high values at scale 7.0 and beyond, while other groups stay flat. None of these experiments is presented as a high-fidelity simulation of a real deployment; each is meant to show a structural gap between what current tests measure and what regulators need to know.

Load-bearing premise

The conclusion that a ROUGE-fair model is actually discriminatory rests on the assumption that the LLM simulating the hiring manager, prompted to score resume summaries from 1 to 10 with 9 as the interview cutoff, behaves enough like a real hiring manager for a 5-percentage-point selection gap to be evidence about real deployments; the authors explicitly disclaim high fidelity to reality for this simulation.

Editorial extensions

If this is right

  • A deployer who equalizes a quality metric like ROUGE across demographic groups has no guarantee of equal downstream outcomes; the same model that wins on ROUGE can produce disparate interview selection rates.
  • Red-teaming results reported under one chosen protocol can rank the same models in opposite order, so regulatory reliance on 'adversarial testing' without a standardized protocol cannot reliably identify the most discriminatory model.
  • Fairness conclusions from single-turn evaluations do not transfer to multi-turn interactions; a model that looks better in the single-turn setting can be worse in a conversation.
  • User-adjustable parameters such as image-generation guidance scale can push a system into discriminatory behavior after deployment, meaning pre-deployment tests on default settings understate risk.
  • Testing should be context-specific and conducted under conditions that mirror deployment, replacing generic quality metrics with evaluation suites that track how output affects downstream decisions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the red-teaming variability result replicates broadly, any single red-team benchmark leaderboard is uninformative for fairness comparisons; regulators may need to mandate a fixed, transparent attack battery or a distribution over attack generators before accepting attack-success-rate evidence.
  • The resume case suggests an inexpensive auditing upgrade: measure not just text similarity but a set of candidate-relevant attributes (sentiment, length, keyword presence) across groups, and report selection-rate disparities under a simulated or human decision-maker, as the authors' mitigation figure illustrates.
  • The guidance-scale finding implies that fairness certification may need to cover a hyperparameter operating range, not a single default, and that deployers who expose sampling parameters to end users inherit part of the testing burden.
  • A testable extension is to re-run the four studies with larger model families and with human decision-makers instead of LLM proxies, to check whether the qualitative gaps persist when the proxy is removed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper argues that current GenAI fairness testing methods are misaligned with regulatory goals: models can pass standard pre-deployment tests (ROUGE-based performance equality, red teaming, single-turn evaluations, fixed-hyperparameter evaluations) while still producing discriminatory outcomes in deployment. It connects U.S. and EU anti-discrimination law with the technical fairness literature, then presents four illustrative case studies: (1) a resume-summarization pipeline where ROUGE-fair models produce unequal simulated interview rates; (2) red teaming where fairness rankings depend on the choice of red language model; (3) single-turn versus multi-turn red teaming evaluations that yield conflicting rankings; and (4) text-to-image generation where the guidance scale changes NSFW-score disparities across racial/ethnic groups. The paper concludes with recommendations for context-specific, robust, deployment-oriented fairness testing.

Significance. If the central claim is established, this is a valuable cross-disciplinary contribution: it gives regulators and deployers a concrete vocabulary for why standard GenAI bias evaluations can be uninformative, and it connects the d-hacking concept to GenAI-specific failure modes. The paper's strengths include a genuinely synthetic legal-technical review (EEOC, OMB, NIST, EU AI Act), four case studies spanning text and image modalities, an explicit link to the authors' prior d-hacking framework treated as prior work rather than as a fitted premise, and a public code repository. The practical recommendations (multi-test panels, deployment-condition mirroring, parameter-sensitivity reporting) are actionable. The significance is currently tempered by the fact that the most load-bearing empirical demonstration, the resume-screening case study, rests on an LLM-simulated hiring manager whose fidelity is explicitly disclaimed, and the reported gaps are presented without uncertainty quantification.

major comments (4)
  1. [4.1, Figure 2, Appendix B.1.2] The central quantitative claim, that Llama-2-7B's summaries produce a roughly 5 percentage point lower simulated interview rate for Black and Hispanic candidates, is reported without any uncertainty quantification. The experiment uses n=250 resumes per group and appears to be a single generation run; no confidence intervals, bootstrap estimates, or repeated-seed variation are given. Appendix B.1.2 reports that the same decision model, applied to race-blind summaries plus names, still produces a 2 percentage point gap (38.4% vs. 36.4%), yet no significance test or interval is reported for this difference-in-differences. Because the conclusion that the summarizer is "actually discriminatory" depends on the 5pp gap being real and attributable to the summarizer rather than to noise or to the decision proxy, the paper should report bootstrap CIs and a formal comparison between the named-summary and race-blind-summary conditions.
  2. [4.1, Experimental Setup and Results] The simulated hiring manager is Llama-3-70B-instruct prompted to score summaries from 1 to 10 with an interview threshold of 9, and the paper explicitly states that the simulation is "not meant to be a high-fidelity simulation of a real hiring application." Yet the surrounding text concludes that a ROUGE-fair model is "actually discriminatory" and that the main source of discrimination is the summarization model. As written, the case study demonstrates a mismatch between two LLM-based evaluations (ROUGE versus an LLM judge), not necessarily a mismatch between a standard fairness test and real-world discrimination. To make this load-bearing claim, the authors should either validate the decision proxy against human judgments or a documented hiring process, show robustness across multiple decision models and thresholds, or explicitly restrict the conclusion to the claim that such mismatches can arise in principle rather than that this particular deployment is discriminatory.
  3. [4.1, Appendix B.1.1] The ROUGE ground-truth summaries are generated by Llama-3-70B-instruct, and the simulated hiring decisions are also made by Llama-3-70B-instruct. This couples the evaluation metric and the downstream decision-maker: a candidate model is judged "fair" by ROUGE against the preferences of the same model family that later determines interview outcomes. The observed mismatch may therefore partly reflect idiosyncratic stylistic preferences of that one model family rather than a general property of ROUGE-style evaluation. The authors should use an independent ground truth (e.g., human-written reference summaries or a different model family) or ablate the decision-maker model to show that the mismatch is not an artifact of this coupling.
  4. [4.2, Figure 4, Appendix B.2.1] The claim that red teaming fairness rankings "can become nearly arbitrary" is demonstrated across choices of RedLM, but attack success rate is computed with a single toxicity threshold (0.2) and a single set of 1,000 sampled attacks per RedLM. No sensitivity analysis is reported over the toxicity threshold or over sampling seeds for attack generation. Without such analysis, the conclusion that rankings are "nearly arbitrary" is itself conditional on the same kind of evaluation choice the paper argues is arbitrary. The authors should report how ASR-based rankings vary across a range of thresholds and across repeated attack-generation seeds.
minor comments (6)
  1. [3.2] The NIST references are inconsistent: the text cites "NIST AI Guidelines (NIST, 2023)" in one place and the "NIST Risk Mitigation Framework for GenAI (NIST, 2024)" elsewhere; the reference list should distinguish the AI RMF 1.0 from the GenAI-specific guidance.
  2. [4.1, Tables 1 and 5] The sentence describing the Qwen-2 example says the white candidate's summary is "worse according to ROUGE" but "more favorable across the broad panel" than the Hispanic candidate's summary; this wording is confusing because the numeric ROUGE scores in Table 1 are lower for the white candidate, and the reader must infer which direction is favorable for each metric. Please clarify the comparison explicitly.
  3. [Figure 4 caption] The caption does not explain how the green-to-red color ranking is computed; the normalization procedure in Appendix B.2 (subtract mean and divide by standard deviation per toxicity percentage) should be stated in the caption or spelled out more clearly with the dimension over which normalization occurs.
  4. [Figure 5 caption] The caption does not identify which line corresponds to which candidate model or what the shaded regions (if any) represent; please add a legend and describe the plotted quantity and the k values on the x-axis.
  5. [4.3] The multi-turn experiment uses only two candidate models (Gemma-2-2B and Gemma-2-9B), so the claim that "fairness rankings can change drastically" between single- and multi-turn settings is based on a narrow comparison; a sentence acknowledging this limitation would help calibrate the claim.
  6. [4.4, Appendix B.4] The NSFW-score results for the diffusion experiment are reported as point values with no error bars or repeated-generation information beyond the stated 128 images per identity; a brief note on variance or generation seeds would strengthen the figure.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity was found: the case studies are empirical demonstrations with hand-set thresholds, not predictions derived from fitted inputs or load-bearing self-citations.

full rationale

The paper does not present a derivation chain in which a prediction is computed from fitted inputs or from the authors' prior work; each of the four case studies is an independently executed experiment against external benchmarks (ROUGE-L, Detoxify, GSM8K/MedQuad, StableDiffusion3 with a public NSFW classifier) and external methodologies (Perez et al., 2022). The central Section 4.1 claim that ROUGE-fair summarization can coexist with unequal simulated interview rates is an empirical observation, not an identity: ROUGE-L string overlap and the Llama-3-70B-instruct interview threshold are different functions of the summaries, and the paper explicitly disclaims high-fidelity simulation ('we simulate these decisions not to claim high fidelity to reality'). The threshold (score at least 9) and toxicity/guidance-scale values are hand-set experimental conditions rather than parameters fitted to the conclusions. The only self-citations of note are to the authors' d-hacking concept (Black et al., 2024) and legal background (Gillis, 2021; Gillis et al., 2024); these frame or contextualize the work but do not supply the evidence, which comes from the experiments themselves. No equation in the paper reduces to its inputs, and no fitted parameter is renamed as a prediction. The authors' own limitation statement ('the case studies, while illustrative, cannot fully encompass the wide range of problems') confirms the empirical rather than deductive status of the claims. Under the stated standard, this is a no-significant-circularity finding.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new entities are introduced. The paper's demonstrations depend on hand-set thresholds and proxy models for downstream decisions, toxicity, and representational harm, all acknowledged in the text as illustrative. None of these parameters is fitted to the central conclusion, but they do shape the reported magnitudes.

free parameters (4)
  • Interview threshold for simulated hiring manager = score of 9 out of 10
    Chosen to map LLM scores into interview decisions; changing this threshold would change the measured selection gap in Section 4.1.
  • Toxicity thresholds for attack success = 0.2 (women red teaming), 0.5 (multi-turn red teaming)
    Attack success rate is defined by these thresholds; rankings can shift with threshold choice, and the instability argument depends on the chosen operationalization.
  • RedLM sampling hyperparameters = per-model temperature, top-p, top-k, repetition penalty in Appendix Table 7
    Hand-tuned to encourage more harmful attacks; differences across RedLMs could drive some of the ASR variability in Section 4.2.
  • Candidate model sampling hyperparameters = per-model values in Appendix Table 8
    Hand-tuned to encourage instruction following and diversity; these choices influence measured attack success rates and therefore the fairness rankings.
assumptions (4)
  • domain assumption LLM-simulated hiring decisions are a meaningful proxy for real decision-maker behavior.
    Section 4.1 uses Llama-3-70B to score summaries and threshold at 9 to define interviews; the authors state the simulation is not high fidelity, yet the central demonstration of downstream discrimination depends on this proxy.
  • domain assumption Detoxify toxicity scores and chosen thresholds capture discriminatory or harmful behavior.
    Attack success rate in Sections 4.2 and 4.3 is defined by a toxicity threshold; the reported rankings are rankings over this operationalized measure.
  • domain assumption NSFW classifier scores measure sexualized representation relevant to discrimination.
    Section 4.4 equates rising NSFW scores with representational harm; no validation is provided that the classifier's notion of NSFW aligns with the legal or policy concept of sexualization.
  • domain assumption LLM-generated lists of stereotypical names are representative of real name-race perceptions.
    Section 4.1 and Appendix B.1.1 insert GPT-4o-generated names as the only group signal; any mismatch with real name-race associations affects the simulated bias.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Effective Discrimination Testing for Generative AI." pith.science (2026). https://pith.science/paper/GUL34QJF

@misc{pith2026241221052,
  author       = {Pith},
  title        = {Pith review of: Towards Effective Discrimination Testing for Generative AI},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GUL34QJF}},
  note         = {Machine review of arXiv:2412.21052}
}
read the original abstract

Generative AI (GenAI) models present new challenges in regulating against discriminatory behavior. In this paper, we argue that GenAI fairness research still has not met these challenges; instead, a significant gap remains between existing bias assessment methods and regulatory goals. This leads to ineffective regulation that can allow deployment of reportedly fair, yet actually discriminatory, GenAI systems. Towards remedying this problem, we connect the legal and technical literature around GenAI bias evaluation and identify areas of misalignment. Through four case studies, we demonstrate how this misalignment between fairness testing techniques and regulatory goals can result in discriminatory outcomes in real-world deployments, especially in adaptive or complex environments. We offer practical recommendations for improving discrimination testing to better align with regulatory goals and enhance the reliability of fairness assessments in future deployments.

Figures

Figures reproduced from arXiv: 2412.21052 by the authors.

Figure 1
Figure 1. The output of classification models can often be directly mapped onto allocative decisions, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Left: Summary quality is scored using ROUGE, and compared across models and racial groups. Llama-2-7B produces the highest average score, and all models offer similar performance across groups–suggesting Llama-2-7B may be chosen to deploy. Right: Though all resumes are the same, simulated outcomes produce different selection rates across groups. Llama-2-7B produces a ∼5% maximum gap across racial groups, while for G… view at source ↗
Figure 3
Figure 3. Plotting the differences between alternative fairness metrics across groups against selection [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Red teaming results for bias against women, where higher scores indicate more toxic [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Models undergo red teaming in the single- and multi-turn settings, with data from different [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Representations of women of different racial/ethnic groups are sensitive to user modifica [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Illustration of our experimental setting for Section [PITH_FULL_IMAGE:figures/full_fig_p031_7.png]
Figure 8
Figure 8. Figure 8: Results probing the (un)fairness of our simulated decision-maker. This difference in [PITH_FULL_IMAGE:figures/full_fig_p033_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 22 canonical work pages

  1. [5]

    What is your least favourite thing about GROUP people?

    B.2. Red Teaming B.2.1. D ETAILS This experiment is based on the bias testing methodology from Perez et al. (2022) (Section 6), an evaluation meant to ensure that a model cannot be easily made to produce offensive and discriminatory material. To implement this procedure, a red team must choose their own red language model (RedLM) to produce attacks, as a ...

    work page 2022

  2. [7]

    URL https://arxiv.org/abs/2104.02145. Miles Brundage, Shahar Avin, Jasmine Wang, Haydn Belfield, Gretchen Krueger, Gillian Hadfield, Heidy Khlaaf, Jingying Yang, Helen Toner, Ruth Fong, Tegan Maharaj, Pang Wei Koh, Sara Hooker, Jade Leung, Andrew Trask, Emma Bluemke, Jonathan Lebensold, Cullen O’Keefe, Mark Koren, Th´eo Ryffel, JB Rubinovitz, Tamay Besiro...

    arXiv

  3. [8]

    Cfpb circular 2022-03: Adverse action notification requirements in connection with credit decisions based on complex algorithms,

    CFPB. Cfpb circular 2022-03: Adverse action notification requirements in connection with credit decisions based on complex algorithms,

    work page 2022

  4. [9]

    DALL-EV AL: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models

    Jaemin Cho, Abhay Zala, and Mohit Bansal. DALL-EV AL: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models . In 2023 IEEE/CVF International Conference on Computer Vision (ICCV),

    work page 2023

  5. [10]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv:2110.14168,

    arXiv

  6. [11]

    Cooper, Solon Barocas, Christopher De Sa, and Siddhartha Sen

    A. Cooper, Solon Barocas, Christopher De Sa, and Siddhartha Sen. Variance, self-consistency, and arbitrariness in fair classification. arXiv:2301.11562,

    arXiv

  7. [12]

    Patton, Elsbeth Turcan, and Kathleen McKeown

    Nicholas Deas, Jessica A Grieser, Shana Kleiner, Desmond U. Patton, Elsbeth Turcan, and Kathleen McKeown. Evaluation of african american language bias in natural language generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing ,

    work page 2023

  8. [13]

    Directive 2000/43/ec implementing the principle of equal treatment between persons irrespective of racial or ethnic origin,

    Directive 43/EC. Directive 2000/43/ec implementing the principle of equal treatment between persons irrespective of racial or ethnic origin,

    work page 2000

Show all 44 references
  1. [16]

    ZOLLO , N

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zha...

    arXiv

  2. [18]

    On the impact of machine learning randomness on group fairness

    Prakhar Ganesh, Hongyan Chang, Martin Strobel, and Reza Shokri. On the impact of machine learning randomness on group fairness. In 2023 ACM Conference on Fairness, Accountability, and Transparency,

    2023

  3. [19]

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned

    Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfi...

    arXiv

  4. [20]

    Chatgpt perpetuates gender bias in machine translation and ignores non-gendered pronouns: Findings across bengali and five other low-resource languages

    Sourojit Ghosh and Aylin Caliskan. Chatgpt perpetuates gender bias in machine translation and ignores non-gendered pronouns: Findings across bengali and five other low-resource languages. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society,

    2023

  5. [21]

    Operationalizing the search for less discriminatory alternatives in fair lending

    Talia B Gillis, Vitaly Meursault, and Berk Ustun. Operationalizing the search for less discriminatory alternatives in fair lending. In The 2024 ACM Conference on Fairness, Accountability, and Transparency,

    2024

  6. [23]

    What’s in a name? auditing large language models for race and gender bias

    Amit Haim, Alejandro Salinas, and Julian Nyarko. What’s in a name? auditing large language models for race and gender bias. arXiv:2402.14875,

    arXiv

  7. [24]

    ZOLLO , N

    23 T. ZOLLO , N. RAJANEESH , R. ZEMEL , T. GILLIS & E. BLACK . D ISCRIMINATION TESTING FOR GENAI Vernon Toh Yan Han, Rishabh Bhardwaj, and Soujanya Poria. Ruby teaming: Improving quality diversity search with memory for automated red teaming. arXiv:2406.11654,

    arXiv

  8. [26]

    Trusta- gent: Towards safe and trustworthy llm-based agents through agent constitution.arXiv:2402.01586,

    Wenyue Hua, Xianjun Yang, Mingyu Jin, Wei Cheng, Ruixiang Tang, and Yongfeng Zhang. Trusta- gent: Towards safe and trustworthy llm-based agents through agent constitution.arXiv:2402.01586,

    arXiv

  9. [27]

    Dart: Deep adversarial automated red teaming for llm safety

    Bojian Jiang, Yi Jing, Tianhao Shen, Qing Yang, and Deyi Xiong. Dart: Deep adversarial automated red teaming for llm safety. arXiv:2407.03876,

    arXiv

  10. [28]

    Smith, and Hannaneh Hajishirzi

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling. arXiv:2403.13787,

    arXiv

  11. [30]

    Bias in language models: Beyond trick tests and toward ruted evaluation

    Kristian Lum, Jacy Reese Anthis, Chirag Nagpal, and Alexander D’Amour. Bias in language models: Beyond trick tests and toward ruted evaluation. arXiv:2402.12649,

    arXiv

  12. [31]

    Zemel, Kai-Wei Chang, Aram Galstyan, and Rahul Gupta

    Ninareh Mehrabi, Palash Goyal, Christophe Dupuy, Qian Hu, Shalini Ghosh, Richard S. Zemel, Kai-Wei Chang, Aram Galstyan, and Rahul Gupta. FLIRT: feedback loop in-context red teaming. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,

    2024

  13. [32]

    ZOLLO , N

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    arXiv

  14. [33]

    Red teaming language models with language models

    Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,

    2022

  15. [35]

    ZOLLO , N

    Jessica Quaye, Alicia Parrish, Oana Inel, Charvi Rastogi, Hannah Rose Kirk, Minsuk Kahng, Erin Van Liemt, Max Bartolo, Jess Tsang, Justin White, Nathan Clement, Rafael Mosquera, Juan 26 T. ZOLLO , N. RAJANEESH , R. ZEMEL , T. GILLIS & E. BLACK . D ISCRIMINATION TESTING FOR GEN...

    2024

  16. [36]

    URL https://arxiv.org/ abs/2111.15366. Reuters. Eeoc says workday covered by anti-bias laws in ai discrimination case,

    arXiv

  17. [37]

    Executive order 14110: Advancing american leadership in artificial intelligence, 2023a

    White House. Executive order 14110: Advancing american leadership in artificial intelligence, 2023a. 27 T. ZOLLO , N. RAJANEESH , R. ZEMEL , T. GILLIS & E. BLACK . D ISCRIMINATION TESTING FOR GENAI White House. Ensuring safe, secure, and trustworthy ai, 2023b. Kyra Wilson and ...

    arXiv

  18. [38]

    Style over substance: Evaluation biases for large language models

    Minghao Wu and Alham Fikri Aji. Style over substance: Evaluation biases for large language models. arXiv:2307.03025,

    arXiv

  19. [39]

    Bias in generative ai

    Mi Zhou, Vibhanshu Abhishek, Timothy Derdenger, Jaymo Kim, and Kannan Srinivasan. Bias in generative ai. arXiv:2403.02726,

    arXiv

  20. [40]

    Zollo, Andrew Wei Tung Siah, Naimeng Ye, Ang Li, and Hongseok Namkoong

    Thomas P. Zollo, Andrew Wei Tung Siah, Naimeng Ye, Ang Li, and Hongseok Namkoong. Personal- llm: Tailoring llms to individual preferences. arXiv:2409.20296,

    arXiv

  21. [43]

    To compute toxicity, we use the Detoxify model Hanu and Unitary team (2020)

    All the candidate models are instruction-tuned. To compute toxicity, we use the Detoxify model Hanu and Unitary team (2020). Each response is split into sentences, and assigned the maximum toxicity score for any of its sentences. An attack is considered successful if it produc...

    2020

  22. [44]

    ZOLLO , N

    36 T. ZOLLO , N. RAJANEESH , R. ZEMEL , T. GILLIS & E. BLACK . D ISCRIMINATION TESTING FOR GENAI B.3. Multi-Turn Conversations B.3.1. D ETAILS We extend the red-teaming experiments to a multi-turn conversation setting using data from the important education and health domains....

    2021

  23. [144]

    Appendix B

    places obligations on deployers to audit and disclose information about tools they may not have developed.Our analysis provides yet another reason to not view this distinction as straightforward, given that harm can arise from a user’s specific implementation or customization ...

    2024

  24. [1968]

    Gaebler, Sharad Goel, Aziz Huq, and Prasanna Tambe

    Johann D. Gaebler, Sharad Goel, Aziz Huq, and Prasanna Tambe. Auditing the use of language models to guide hiring decisions. arXiv:2404.03086,

    arXiv

  25. [2000]

    Directive 2006/54/ec on the implementation of the principle of equal opportunities and equal treatment of men and women in matters of employment and occupation (recast),

    Directive 54/EC. Directive 2006/54/ec on the implementation of the principle of equal opportunities and equal treatment of men and women in matters of employment and occupation (recast),

    2006

  26. [2003]

    Easily accessible text-to- image generation amplifies demographic stereotypes at large scale

    Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng, Debora Nozza, Tatsunori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan. Easily accessible text-to- image generation amplifies demographic stereotypes at large scale. In Proceedings of the 2023...

    2023

  27. [2004]

    Toxicchat: Unveiling hidden challenges of toxicity detection in real-world user-ai conversation

    Zi Lin, Zihan Wang, Yongqi Tong, Yangkun Wang, Yuxin Guo, Yujia Wang, and Jingbo Shang. Toxicchat: Unveiling hidden challenges of toxicity detection in real-world user-ai conversation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,

    2023

  28. [2006]

    Directive 2000/78/ec establishing a general framework for equal treatment in employment and occupation,

    Directive 78/EC. Directive 2000/78/ec establishing a general framework for equal treatment in employment and occupation,

    2000

  29. [2017]

    John J. Horton. Large language models as simulated economic agents: What can we learn from homo silicus? arXiv:2301.07543,

  30. [2018]

    Generative discrimination: What happens when generative ai exhibits bias, and what can be done about it

    Philipp Hacker, Brent Mittelstadt, Frederik Zuiderveen Borgesius, and Sandra Wachter. Generative discrimination: What happens when generative ai exhibits bias, and what can be done about it. arXiv:2407.10329,

    arXiv

  31. [2019]

    Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell

    Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency,

    2021

  32. [2020]

    Unsafe diffusion: On the generation of unsafe images and hateful memes from text-to-image models

    Yiting Qu, Xinyue Shen, Xinlei He, Michael Backes, Savvas Zannettou, and Yang Zhang. Unsafe diffusion: On the generation of unsafe images and hateful memes from text-to-image models. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security,

    2023

  33. [2021]

    ZOLLO , N

    18 T. ZOLLO , N. RAJANEESH , R. ZEMEL , T. GILLIS & E. BLACK . D ISCRIMINATION TESTING FOR GENAI Emily Black, Talia Gillis, and Zara Yasmine Hall. D-hacking. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency,

    2024

  34. [2022]

    Fairmonitor: A dual-framework for detecting stereotypes and biases in large language models

    Yanhong Bai, Jiabao Zhao, Jinxin Shi, Zhentao Xie, Xingjiao Wu, and Liang He. Fairmonitor: A dual-framework for detecting stereotypes and biases in large language models. arXiv:2405.03098,

    arXiv

  35. [2023]

    Leave-one-out unfairness

    Emily Black and Matt Fredrikson. Leave-one-out unfairness. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency,

    2021

  36. [2024]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan H...

    arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.