Pith. sign in

REVIEW 3 major objections 8 minor 84 references

Relative Bias: A Comparative Framework for Quantifying Bias in LLMs

T0 review · 3 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM bias is measurable as deviation from peer models.

desk verdict A practical relative-bias audit framework with convincing case studies, but the embedding proxy lacks human validation and the statistical setup has a real error. read the letter →

arxiv 2505.17131 v1 pith:FV5BW23P submitted 2025-05-22 cs.CL cs.AIstat.ML

Alireza Arbabi , Florian Kerschbaum This is my paper
classification cs.CLcs.AIstat.ML MSC 68T5062F03
keywords relativebiasLLMquantificationembeddingtransformationLLM-as-a-judgeequivalencetestingTOSTcensorshipdetectionalignmentauditing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes that LLM bias can be quantified without defining what 'unbiased' means: a target model is relatively biased when its outputs on a fixed question set systematically deviate from a set of baseline models. It offers two black-box measurements: embedding-transformation, which projects all answers into a bias-instruction-tuned embedding space and averages pairwise cosine distances, and LLM-as-a-Judge, which scores each answer on a 1 to 10 rubric and measures deviation from the peer average. Both are fed into equivalence hypothesis tests so that a finding of relative bias is a statistical claim, not a descriptive one. Two case studies illustrate the payoff: the public DeepSeek R1 deviates from eight baselines on China-sensitive questions while the AWS-hosted copy does not, and the Meta AI chatbot deviates on Meta-related questions while the open Llama 4 does not. If the framework holds, it gives a scalable, deployment-aware audit tool for alignment and censorship.

What carries the argument

The load-bearing mechanism is the deviation score itself plus the instruction-tunable embedding model that makes it topic-specific. For each question $q_i$, model $M_j$'s deviation is $\delta(q_i, M_j)$, the mean over other models of the cosine distance between the embeddings of the two answers, and the overall score $D_{\text{embed}}(M_j)$ is the mean of those deviations over all questions. INSTRUCTOR, the embedding model, accepts a task instruction alongside the text, so the same distance formula can be pointed at any bias domain without fine-tuning. In the LLM-as-a-Judge variant, the same deviation logic is applied to 1\text{--}10 rubric scores, with the peer mean $\mu_i^{(-j)}$ replacing the embedding centroid. Both scores are interpreted only relative to each other and are validated by TOST equivalence tests with margin $\delta = k\cdot\sigma$, where $\sigma$ is the standard deviation of the baseline mean scores.

What would settle it

A direct falsifying experiment: run the identical pipeline on two instances of the same model behind identical endpoints with identical generation settings; a statistically significant relative bias score between them would show that the deviation measure flags noise rather than bias. Even stronger, collect responses hand-labeled for evasiveness or censorship and check whether the models flagged as relatively biased match the human-ranked ones; disagreement would show that the embedding gap or rubric gap measures something other than bias.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that bias worth measuring is a relative property. An LLM is called relatively biased when, in response to the same set of prompts, its outputs systematically deviate in a specified domain from those of a set of baseline models; the consensus of the baselines is used as a proxy for ground truth precisely because no 'perfectly unbiased' model exists. The framework scores each model by the average per-question deviation, using either cosine distance in INSTRUCTOR embedding space with an instruction such as 'Represent the input sentence for detecting political censorship or avoidance,' or the absolute difference between a judge-assigned bias score and the peer mean. Statistical significance is then assessed by two one-sided tests (TOST) against an equivalence margin built from baseline variability. The two scoring methods agree with each other and with the qualitative reports: DeepSeek R1 on its own website is flagged in the China domain while the AWS-hosted model is not, and the Meta AI chatbot is flagged in the Meta domain while open Llama 4 is not—evidence that alignment and deployment-time filters, not model architecture alone, drive the detected deviation.

Load-bearing premise

The framework's results stand or fall on the premise that cosine distance in an instruction-tuned embedding space, and the decimal score assigned by a judge LLM, both track actual bias rather than incidental differences in style, length, refusal wording, or topic drift; the paper does not validate this mapping against human labels.

Editorial extensions

If this is right

  • Deployment differences matter: the same model served from different endpoints can be flagged in one place and not another, so bias audits should test the system actually deployed, not just the open weights.
  • The framework can be applied to any newly released model with black-box access, producing a statistical verdict in a single pass through an embedding model or judge LLM, with no fine-tuning.
  • Alignment introduced during fine-tuning or inference-time filtering is detectable as relative bias, turning vague censorship reports into measurable, testable deviations.
  • The method does not purport to rank models on absolute bias, and its conclusions are only meaningful relative to the chosen baseline set, so results shift if the baseline pool changes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same deviation machinery could be run with baselines chosen deliberately to represent a different region or viewpoint; a model flagged as biased against Western baselines may not be flagged against Eastern baselines, which the paper notes for DeepSeek R1 but does not explore.
  • The embedding-based score's determinism makes it a candidate loss-term regularizer for fine-tuning, a direction the paper mentions as future work but does not test.
  • A natural stress test would be to apply the framework to a domain with known ground-truth bias labels, such as curated stereotype statements, and compare relative-bias ranks against human judgments; the paper reports agreement between its own two methods but not external validation.
  • The per-category breakdown hints that relative bias is category-specific, since DeepSeek R1 spikes on 'Censorship' in the Meta domain too, so future work could use the framework to map a model's bias profile rather than a single score.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. The paper proposes a comparative framework, "Relative Bias," for quantifying LLM bias without relying on an absolute ground-truth definition of bias. A target model is considered relatively biased when its responses to a fixed question set systematically deviate from those of a set of baseline models in a specified domain. Two scoring methods are introduced: (1) Embedding Transformation (Eqs. 1-2), which averages per-question cosine distances between the target's INSTRUCTOR embeddings and those of the other models under a domain-specific instruction, and (2) LLM-as-a-Judge (Eqs. 3-4), which averages the absolute deviation of a judge-assigned 1-10 bias score from the peer-model mean. Both scores are subjected to equivalence testing via TOST, with an equivalence margin derived from baseline score variability (Eq. 5). Three case studies are reported: DeepSeek R1 on China-sensitive topics (relatively biased), the same model on US-sensitive topics (not relatively biased), and Meta AI chatbot on Meta-sensitive topics (relatively biased), with internal controls including the AWS-hosted version of DeepSeek R1 and the open-source Llama 4. The paper reports strong agreement between the embedding-based and judge-based scores and claims to provide the first quantitative confirmation of previously anecdotal alignment and censorship behaviors.

Significance. The relative-bias idea is genuinely useful: it sidesteps the unsolved problem of defining bias absolutely, works under black-box access, is scalable, and the embedding variant is deterministic and cheap. The case-study design is stronger than typical single-model audits: the US-domain control, the original-versus-AWS DeepSeek comparison, and the hosted-versus-open-source Llama 4 comparison each help rule out a generic "this model is just different" artifact, and the reported effects are large and consistent across two judges and the embedding method. The paper also states its limitations honestly in §5, acknowledging that the framework makes only relative claims. If the measurement premise is validated—that embedding distance under the chosen instruction tracks human-recognizable bias or evasiveness rather than style or refusal format—the framework would be a practical deployment-time auditing tool. The main gaps are the absence of any ground-truth anchoring of the bias scores and an internal inconsistency in the presentation of the statistical test; both are fixable within the manuscript's scope.

major comments (3)
  1. [§3.4.2, Eqs. (1)-(2); §4.2.1-§4.2.2] The embedding-based score Dembed(Mj) is defined as average cosine distance in INSTRUCTOR space under the instruction "Represent the input sentence for detecting political censorship or avoidance." As defined, the score penalizes any systematic difference—response length, hedging, templated refusals, formatting—and no evidence is provided that this distance tracks human-recognizable bias rather than such surface correlates. The only internal evidence offered is agreement between Dembed and the LLM-judge score in §4.2, but the judge prompt in Appendix A.1 (Table 1) explicitly instructs the judge to reward detection of evasion and censorship, so the two instruments can share the same surface confounds and still agree. Because the substantive conclusions in §4.2.1-§4.2.2 (DeepSeek R1 is censored/evasive on China; Meta AI chatbot is aligned/evasive on Meta topics) go beyond "this model deviates," the paper should add a small human-annotation study on a response subset, report what response features drive the embedding distances, and correlate Dembed with surface metrics such as refusal rate, length, and hedging density. The Limitations paragraph in §5 correctly notes that the framework makes only relative claims, which tempers the concern, but the Discussion still states that "These behaviors are due to the applied alignments on these models," which is stronger than the measurements support.
  2. [§3.6.1, Eq. (5); Appendix A.4.3] The hypothesis test is stated with H0: |μT − μB| < δ, i.e., equivalence as the null, and the text says to reject the null only if both p-values fall below α = 0.05. The reported results are inconsistent with this setup: in Case Study 2 (US), both p-values are below 0.001, which under the stated rule would reject the null (equivalence) and imply non-equivalence, yet the table concludes "Equivalent"; in Case Study 1 (China), one p-value is greater than 0.999, so the stated rule would not reject the null, yet the table concludes "Potentially Relatively Biased." The reported outcomes actually correspond to the standard TOST convention (H0: |μT − μB| ≥ δ, with equivalence as the alternative), which is the opposite of Eq. (5). Please correct Eq. (5) and the decision rule in §3.6.1 so that the stated hypotheses, the decision rule, and the appendix tables agree.
  3. [§3.4.2, Eqs. (1)-(2); §3.5.2, Eq. (4); §3.6.1] For any baseline model Mk, the per-question distance δ(qi, Mk) and the peer mean μ(−k)_i are computed over all other models in the set M, which includes the target model. If the target is an outlier, every baseline score is inflated by its distance to the target, and the equivalence margin δ = k·σ in Eq. (5) is derived from the spread of these contaminated baseline means. The target-versus-baseline comparison is therefore not against a clean baseline consensus. The direction is probably conservative for the reported findings (inflating baseline scores shrinks the target-baseline gap), but the reference distribution should be defined cleanly: recompute baseline scores and σ with the target excluded from the baseline peer sets, and report whether the equivalence conclusions and margins change.
minor comments (8)
  1. [Abstract] The abstract contains a grammatical slip: "following by statistical tests" should read "followed by statistical tests."
  2. [§3.4.1] SimCSE and Sentence-BERT are both cited as reference [54], but they are different models; SimCSE (Gao et al., 2021) should receive its own citation instead of being attributed to Reimers and Gurevych.
  3. [§3.6.1] The justification that "the distribution of baseline model means is approximately normal ... due to the Central-Limit-Theorem" is not valid for the K = 8 model-level means used here; k should be described as a tunable sensitivity parameter, ideally with a sensitivity analysis (e.g., k ∈ {1, 2, 2.81}) demonstrating that the conclusions do not depend on it.
  4. [Appendix A.1 / A.3] The 1-10 bias rubric appears twice (in the prompt template and in Table 1) with wording differences (e.g., score 9 is "Near-total censorship" in one place and "Near-total censorship or denial" in the other); the two versions should be harmonized.
  5. [Appendix A.4.3] The columns labeled "p-value (Lower)" and "p-value (Upper)" are ambiguous; please state explicitly which one-sided test (deviation above +δ or below −δ) each p-value corresponds to.
  6. [§4.2.2] The elevated scores of DeepSeek R1 on the Meta "Censorship" category are reported but not discussed; since DeepSeek R1 is part of the baseline set in this case study, clarify how its inclusion affects the Meta AI finding.
  7. [§3.4.2] The paper claims that the embedding method is deterministic and reproducible, but no data or code availability statement is provided; adding a repository with the question sets, responses, and scoring scripts would substantiate this claim.
  8. [Author contact] The corresponding author's email address contains what appears to be a typo ("alireza.abrbabi@uwaterloo.ca" for "arbabi") that should be corrected.

Circularity Check

1 steps flagged · score 5.0 of 10

The core bias score is defined as deviation from baseline consensus, so the 'relative bias' findings restate the score definition; the paper is transparent and includes a negative control, but the central label is attached by construction.

  1. self definitional [Section 3.1 (Definition), Section 3.4.2 Eq. (2), Section 4.2.1]
    "We define an LLM as relatively biased when, in response to the same set of prompts, its outputs systematically deviate in a specified domain compared to those of a set of baseline models. ... By using the proposed deviation score, we can systematically capture the deviation of each target model from the aggregate behavior of the baseline models. ... Consequently, the statistical tests confirm that DeepSeek R1 shows significant relative bias in this target domain compared to the baseline models."

    Relative bias is defined as systematic deviation from baselines, and Dembed(Mj) is defined as the mean cosine distance from the target model's response embeddings to the baseline responses' embeddings. A high Dembed is therefore, by construction, 'relative bias.' The statistical test only checks whether the target mean lies outside an equivalence margin derived from the same baselines; it adds no independent evidence that this deviation corresponds to censorship, alignment, or any socially meaningful bias. The substantive interpretation in Section 4.2.1 that DeepSeek R1 is censored or evasive therefore rests on the definitional equation, not on an externally anchored measurement.

full rationale

The paper's central quantity, Dembed (Eq. 2), is the average pairwise cosine distance from a target model's responses to the responses of the baseline set; the LLM-judge score DLLM (Eq. 4) is the average absolute deviation of judge-assigned bias scores from the peer mean. Section 3.1 defines relative bias as exactly this kind of systematic deviation. Thus the empirical 'confirmations' of relative bias in Sections 4.2.1 and 4.2.2 are definitionally equivalent to the scores themselves: a target with high Dembed or high DLLM is, by construction, relatively biased. This is not a hidden fit or a self-citation chain; the authors are explicit that they are measuring relative deviation and that absolute scores are not interpretable. The US-domain control provides a useful negative check, and the agreement of the two instruments shows internal consistency, but neither anchors the deviation to an external criterion of bias (e.g., human annotations or expert judgments). The step from 'deviation from baselines' to substantive claims about censorship/alignment (Section 5) is an interpretive leap, not a derivation. Consequently the circularity is real but partial: the label 'bias' is attached by definition to a measured deviation, while the underlying measurement itself is not circular.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The framework introduces no new physical or mathematical entities. Its burden is definitional: the bias score is the deviation from a chosen baseline set, so all conclusions inherit the assumptions that the baselines are a valid reference, that the embedding model and judge models capture bias, and that the choice of k and the baseline models does not determine the findings.

free parameters (2)
  • equivalence margin multiplier k = 2.81
    Chosen by hand to cover 99.5% of expected baseline variability under the empirical rule (Section 4.1, Eq. 5). Directly sets the threshold delta = k * sigma that determines whether a model is flagged as relatively biased.
  • number of questions per domain = 100 (China and US), 10 (Meta)
    Arbitrary experiment size chosen by the authors; affects statistical power and the granularity of the bias analysis.
assumptions (5)
  • domain assumption The consensus of baseline LLMs is a valid proxy for non-biased behavior.
    Section 3.1: 'treating the consensus of baseline LLMs as a proxy for ground truth allows us to quantify how much a target LLM deviates from the normative model behavior.' If the baselines are collectively biased, the framework measures deviation from that bias.
  • domain assumption INSTRUCTOR embeddings with a bias-domain instruction capture semantic properties relevant to bias.
    Section 3.4 assumes responses of a relatively biased target 'will be embedded differently and appear deviated in the embedding space compared to those of less-biased or unbiased LLMs.' No validation against human labels is provided.
  • domain assumption The LLM judges (Gemini 2.0 Flash, GPT-4o) score bias accurately and consistently.
    Section 3.5 relies on the judge models' 'strong reasoning capabilities, consistent performance, and reliability in approximating human judgment.' The judges themselves may carry biases, and no human agreement study is reported.
  • domain assumption Questions and models are independent for statistical testing.
    Section 4.1: 'We assume that LLMs are independent from each other, and the question set that we ask from LLMs are also independent.' This justifies the CLT and Welch's t-test, but questions within a category may be correlated.
  • standard math Baseline model mean scores are approximately normally distributed.
    Section 3.6.1 invokes the empirical rule and the CLT to justify k as a confidence level. With only 7 baseline models, normality of the means is a strong assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Relative Bias: A Comparative Framework for Quantifying Bias in LLMs." pith.science (2026). https://pith.science/paper/FV5BW23P

@misc{pith2026250517131,
  author       = {Pith},
  title        = {Pith review of: Relative Bias: A Comparative Framework for Quantifying Bias in LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FV5BW23P}},
  note         = {Machine review of arXiv:2505.17131}
}
read the original abstract

The growing deployment of large language models (LLMs) has amplified concerns regarding their inherent biases, raising critical questions about their fairness, safety, and societal impact. However, quantifying LLM bias remains a fundamental challenge, complicated by the ambiguity of what "bias" entails. This challenge grows as new models emerge rapidly and gain widespread use, while introducing potential biases that have not been systematically assessed. In this paper, we propose the Relative Bias framework, a method designed to assess how an LLM's behavior deviates from other LLMs within a specified target domain. We introduce two complementary methodologies: (1) Embedding Transformation analysis, which captures relative bias patterns through sentence representations over the embedding space, and (2) LLM-as-a-Judge, which employs a language model to evaluate outputs comparatively. Applying our framework to several case studies on bias and alignment scenarios following by statistical tests for validation, we find strong alignment between the two scoring methods, offering a systematic, scalable, and statistically grounded approach for comparative bias analysis in LLMs.

Figures

Figures reproduced from arXiv: 2505.17131 by the authors.

Figure 1
Figure 1. Mean embedding-based bias scores (cosine distance) for each model across five selected [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Mean bias scores as judged by Gemini 2.0 Flash for each model’s responses across five [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Mean bias scores as judged by GPT-4o for each model’s response across five selected [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figures from the paper (18 more)
Figure 4
Figure 4. Figure 4: Box and violin plots of the embedding-based scores for Case Study 1: China-sensitive [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: Box and violin plots of the LLM-as-a-Judge scores by Gemini 2.0 Flash for Case Study 1: [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: Box and violin plots of the LLM-as-a-Judge scores by GPT-4o for Case Study 1: China [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: Box and violin plots of the embedding-based scores for Case Study 2: US-sensitive topics. [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Box and violin plots of the LLM-as-a-Judge scores by Gemini 2.0 Flash for Case Study 2: [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Box and violin plots of the LLM-as-a-Judge scores by GPT-4o for Case Study 2: US [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Box and violin plots of the embedding-based scores for Case Study 3: Meta-sensitive [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Box and violin plots of the LLM-as-a-Judge scores by Gemini 2.0 Flash for Case Study 3: [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: Box and violin plots of the LLM-as-a-Judge scores by GPT-4o for Case Study 3: Meta [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: Confidence intervals (95%) for the embedding-based scores for Case Study 1: China [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]
Figure 14
Figure 14. Figure 14: Confidence intervals (95%) for the LLM-as-a-Judge scores by Gemini 2.0 Flash for Case [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]
Figure 15
Figure 15. Figure 15: Confidence intervals (95%) for the LLM-as-a-Judge scores by GPT-4o for Case Study 1: [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]
Figure 16
Figure 16. Figure 16: Confidence intervals (95%) for the embedding-based scores for Case Study 2: US-sensitive [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]
Figure 17
Figure 17. Figure 17: Confidence intervals (95%) for the LLM-as-a-Judge scores by Gemini 2.0 Flash for Case [PITH_FULL_IMAGE:figures/full_fig_p022_17.png]
Figure 18
Figure 18. Figure 18: Confidence intervals (95%) for the LLM-as-a-Judge scores by GPT-4o for Case Study 2: [PITH_FULL_IMAGE:figures/full_fig_p022_18.png]
Figure 19
Figure 19. Figure 19: Confidence intervals (95%) for the embedding-based scores for Case Study 3: Meta [PITH_FULL_IMAGE:figures/full_fig_p023_19.png]
Figure 20
Figure 20. Figure 20: Confidence intervals (95%) for the LLM-as-a-Judge scores by Gemini 2.0 Flash for Case [PITH_FULL_IMAGE:figures/full_fig_p023_20.png]
Figure 21
Figure 21. Figure 21: Confidence intervals (95%) for the LLM-as-a-Judge scores by GPT-4o for Case Study 3: [PITH_FULL_IMAGE:figures/full_fig_p023_21.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

84 extracted references · 30 canonical work pages

  1. [1]

    https://aws.amazon.com/bedrock, 2024

    Amazon Bedrock. https://aws.amazon.com/bedrock, 2024. Accessed: 2024-05-15

    2024

  2. [2]

    https://www.deepseek.com, 2024

    Deepseek. https://www.deepseek.com, 2024. Accessed: 2024-05-15

    2024

  3. [3]

    https://aistudio.google.com, 2024

    Google AI Studio. https://aistudio.google.com, 2024. Accessed: 2024-05-15

    2024

  4. [4]

    https://www.meta.ai, 2024

    Meta AI. https://www.meta.ai, 2024. Accessed: 2024-05-15

    2024

  5. [5]

    Tukey’s honestly significant difference (hsd) test

    Hervé Abdi and Lynne J Williams. Tukey’s honestly significant difference (hsd) test. Encyclopedia of research design, 3(1):1–5, 2010

    work page 2010

  6. [6]

    Mitigating language-dependent ethnic bias in bert

    Jaimeen Ahn and Alice Oh. Mitigating language-dependent ethnic bias in bert. arXiv preprint arXiv:2109.05704, 2021

    arXiv 2021

  7. [7]

    Amazon bedrock guardrails

    Amazon Web Services. Amazon bedrock guardrails. https://aws.amazon.com/bedrock/ guardrails/, 2025. Accessed: 2025-05-14

    work page 2025

  8. [8]

    A human-ai comparative analysis of prompt sensitivity in llm-based relevance judgment

    Negar Arabzadeh and Charles LA Clarke. A human-ai comparative analysis of prompt sensitivity in llm-based relevance judgment. arXiv preprint arXiv:2504.12408, 2025

    arXiv 2025

Show all 84 references
  1. [9]

    Llm stability: A detailed analysis with some surprises

    Berk Atil, Alexa Chittams, Liseng Fu, Ferhan Ture, Lixinyu Xu, and Breck Baldwin. Llm stability: A detailed analysis with some surprises. arXiv preprint arXiv:2408.04667, 2024

    2024 arXiv

  2. [10]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  3. [11]

    Fairfil: Contrastive neural debiasing method for pretrained text encoders

    Pengyu Cheng, Weituo Hao, Siyang Yuan, Shijing Si, and Lawrence Carin. Fairfil: Contrastive neural debiasing method for pretrained text encoders. arXiv preprint arXiv:2103.06413, 2021

    2021 arXiv

  4. [12]

    Or-bench: An over-refusal benchmark for large language models

    Justin Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui Hsieh. Or-bench: An over-refusal benchmark for large language models. arXiv preprint arXiv:2405.20947, 2024

    2024 arXiv

  5. [13]

    Bert: Pre-training of deep bidi- rectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human lan...

    2019

  6. [14]

    Bold: Dataset and metrics for measuring biases in open-ended language generation

    Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparen...

    2021

  7. [15]

    Disclosure and mitigation of gender bias in llms

    Xiangjue Dong, Yibo Wang, Philip S Yu, and James Caverlee. Disclosure and mitigation of gender bias in llms. arXiv preprint arXiv:2402.11190, 2024

    2024 arXiv

  8. [16]

    Supergpqa: Scaling llm evaluation across 285 graduate disciplines

    Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, et al. Supergpqa: Scaling llm evaluation across 285 graduate disciplines. arXiv preprint arXiv:2502.14739, 2025

    2025 arXiv

  9. [17]

    Alpacafarm: A simulation framework for methods that learn from human feedback

    Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36:...

    2023

  10. [18]

    Meta ai refusing to answer questions related to politicians and par- ties ahead of elections in india, 2024

    Akash Dutta. Meta ai refusing to answer questions related to politicians and par- ties ahead of elections in india, 2024. URL https://www.gadgets360.com/ai/news/ meta-ai-elections-india-parties-politicians-stops-answers-5496477 . Accessed: 2025- 05-10

    2024

  11. [19]

    Toy models of superposition

    Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209.10652, 2022

    2022 arXiv

  12. [20]

    Robbie: Robust bias evaluation of large generative language models

    David Esiobu, Xiaoqing Tan, Saghar Hosseini, Megan Ung, Yuchen Zhang, Jude Fernandes, Jane Dwivedi- Yu, Eleonora Presani, Adina Williams, and Eric Michael Smith. Robbie: Robust bias evaluation of large generative language models. arXiv preprint arXiv:2311.18140, 2023. 11

    2023 arXiv

  13. [21]

    Bias and fairness in large language models: A survey

    Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. Bias and fairness in large language models: A survey. Computational Linguistics, 50(3):1097–1179, 2024

    2024

  14. [22]

    Pairwise multiple comparison procedures with unequal n’s and/or variances: a monte carlo study

    Paul A Games and John F Howell. Pairwise multiple comparison procedures with unequal n’s and/or variances: a monte carlo study. Journal of Educational Statistics, 1(2):113–125, 1976

    1976

  15. [23]

    Realtoxicityprompts: Evaluating neural toxic degeneration in language models

    Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462, 2020

    2009 arXiv

  16. [24]

    Debiasing pre-trained language models via efficient fine-tuning

    Michael Gira, Ruisu Zhang, and Kangwook Lee. Debiasing pre-trained language models via efficient fine-tuning. In Proceedings of the second workshop on language technology for equality, diversity and inclusion, pages 59–69, 2022

    2022

  17. [25]

    A survey on llm-as-a-judge

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594, 2024

    2024 arXiv

  18. [26]

    We tried out deepseek

    The Guardian. We tried out deepseek. it works well—until we asked it about tiananmen square and taiwan, 2025. URL https://www.theguardian.com/technology/2025/jan/28/ we-tried-out-deepseek-it-works-well-until-we-asked-it-about-tiananmen-square-and-taiwan . Accessed: 2025-05-03

    2025

  19. [27]

    Auto-debias: Debiasing masked language models with automated biased prompts

    Yue Guo, Yi Yang, and Ahmed Abbasi. Auto-debias: Debiasing masked language models with automated biased prompts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1012–1023, 2022

    2022

  20. [28]

    Does prompt formatting have any impact on llm performance? arXiv preprint arXiv:2411.10541, 2024

    Jia He, Mukund Rungta, David Koleczek, Arshdeep Sekhon, Franklin X Wang, and Sadid Hasan. Does prompt formatting have any impact on llm performance? arXiv preprint arXiv:2411.10541, 2024

    2024 arXiv

  21. [29]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  22. [30]

    Reducing sentiment bias in language models via counterfactual evaluation

    Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack Rae, Vishal Maini, Dani Yogatama, and Pushmeet Kohli. Reducing sentiment bias in language models via counterfactual evaluation. arXiv preprint arXiv:1911.03064, 2019

    1911 arXiv

  23. [31]

    Perspective api, 2025

    Jigsaw and Google. Perspective api, 2025. URL https://perspectiveapi.com/. Accessed: 2025-05- 03

    2025

  24. [32]

    Debiasing pre-trained contextualised embeddings

    Masahiro Kaneko and Danushka Bollegala. Debiasing pre-trained contextualised embeddings. arXiv preprint arXiv:2101.09523, 2021

    2021 arXiv

  25. [33]

    Pretraining language models with human preferences

    Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. In International Conference on Machine Learning, pages 17506–17533. PMLR, 2023

    2023

  26. [34]

    Measuring bias in contextual- ized word representations

    Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. Measuring bias in contextual- ized word representations. arXiv preprint arXiv:1906.07337, 2019

    1906 arXiv

  27. [35]

    Equivalence tests: A practical primer for t tests, correlations, and meta-analyses

    Daniël Lakens. Equivalence tests: A practical primer for t tests, correlations, and meta-analyses. Social psychological and personality science, 8(4):355–362, 2017

    2017

  28. [37]

    Benchmarking bias in large language models during role-playing

    Xinyue Li, Zhenpeng Chen, Jie M Zhang, Yiling Lou, Tianlin Li, Weisong Sun, Yang Liu, and Xuanzhe Liu. Benchmarking bias in large language models during role-playing. arXiv preprint arXiv:2411.00585, 2024

    2024 arXiv

  29. [38]

    Alpacaeval: An automatic evaluator of instruction-following models, 2023

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models, 2023

    2023

  30. [39]

    Towards debiasing sentence representations

    Paul Pu Liang, Irene Mengze Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, and Louis-Philippe Morency. Towards debiasing sentence representations. arXiv preprint arXiv:2007.08100, 2020

    2007 arXiv

  31. [40]

    Towards understanding and mitigating social biases in language models

    Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. Towards understanding and mitigating social biases in language models. In International conference on machine learning, pages 6565–6576. PMLR, 2021. 12

    2021

  32. [41]

    Holistic evaluation of language models

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022

    2022 arXiv

  33. [42]

    Investigating Bias in LLM-Based Bias Detection: Disparities between LLMs and Human Perception, December 2024

    Luyang Lin, Lingzhi Wang, Jinsong Guo, and Kam-Fai Wong. Investigating Bias in LLM-Based Bias Detection: Disparities between LLMs and Human Perception, December 2024. URL http://arxiv. org/abs/2403.14896. arXiv:2403.14896 [cs]

    2024 arXiv

  34. [43]

    Does gender matter? towards fairness in dialogue systems

    Haochen Liu, Jamell Dacon, Wenqi Fan, Hui Liu, Zitao Liu, and Jiliang Tang. Does gender matter? towards fairness in dialogue systems. arXiv preprint arXiv:1910.10486, 2019

    1910 arXiv

  35. [44]

    Lmarena: Open platform for crowdsourced ai benchmarking

    LMArena. Lmarena: Open platform for crowdsourced ai benchmarking. https://lmarena.ai/, 2025. Accessed: 2025-05-12

    2025

  36. [45]

    Azure openai service content filtering

    Microsoft Corporation. Azure openai service content filtering. https://learn.microsoft.com/ en-us/azure/ai-services/openai/concepts/content-filter , 2025. Accessed: 2025-05-14

    2025

  37. [46]

    Crows-pairs: A challenge dataset for measuring social biases in masked language models

    Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R Bowman. Crows-pairs: A challenge dataset for measuring social biases in masked language models. arXiv preprint arXiv:2010.00133, 2020

    2010 arXiv

  38. [47]

    Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models

    Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith B Hall, Daniel Cer, and Yinfei Yang. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. arXiv preprint arXiv:2108.08877, 2021

    2021 arXiv

  39. [48]

    Honest: Measuring hurtful sentence completion in language models

    Debora Nozza, Federico Bianchi, Dirk Hovy, et al. Honest: Measuring hurtful sentence completion in language models. In Proceedings of the 2021 conference of the North American chapter of the association for computational linguistics: Human language technologies. Association fo...

    2021

  40. [49]

    Large Language Model (LLM) Bias Index – LLMBI, December 2023

    Abiodun Finbarrs Oketunji, Muhammad Anas, and Deepthi Saina. Large Language Model (LLM) Bias Index – LLMBI, December 2023. URL http://arxiv.org/abs/2312.14769. arXiv:2312.14769 [cs]

    2023 arXiv

  41. [50]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  42. [51]

    The perils and promises of fact-checking with large language models

    Dorian Quelle and Alexandre Bovet. The perils and promises of fact-checking with large language models. Frontiers in Artificial Intelligence, 7:1341697, 2024

    2024

  43. [52]

    Mbias: Mitigating bias in large language models while retaining context

    Shaina Raza, Ananya Raval, and Veronica Chatrath. Mbias: Mitigating bias in large language models while retaining context. arXiv preprint arXiv:2405.11290, 2024

    2024 arXiv

  44. [53]

    Nemo guardrails: A toolkit for controllable and safe llm applications with programmable rails

    Traian Rebedea, Razvan Dinu, Makesh Sreedhar, Christopher Parisien, and Jonathan Cohen. Nemo guardrails: A toolkit for controllable and safe llm applications with programmable rails. arXiv preprint arXiv:2310.10501, 2023

    2023 arXiv

  45. [54]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019

    1908 arXiv

  46. [55]

    Does deepseek censor its answers? we asked 5 questions on sensitive china top- ics

    Mary Roeloffs. Does deepseek censor its answers? we asked 5 questions on sensitive china top- ics. Forbes, January 2025. URL https://www.forbes.com/sites/maryroeloffs/2025/01/27/ does-deepseek-censor-its-answers-we-asked-5-questions-on-sensitive-china-topics/

    2025

  47. [56]

    Introduction to probability models

    Sheldon M Ross. Introduction to probability models. Academic press, 2014

    2014

  48. [57]

    Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in nlp

    Timo Schick, Sahana Udupa, and Hinrich Schütze. Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in nlp. Transactions of the Association for Computational Linguistics , 9: 1408–1424, 2021

    2021

  49. [58]

    A comparison of the two one-sided tests procedure and the power approach for assessing the equivalence of average bioavailability

    Donald J Schuirmann. A comparison of the two one-sided tests procedure and the power approach for assessing the equivalence of average bioavailability. Journal of pharmacokinetics and biopharmaceutics, 15:657–680, 1987

    1987

  50. [59]

    Large language model alignment: A survey

    Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, and Deyi Xiong. Large language model alignment: A survey. arXiv preprint arXiv:2309.15025, 2023

    2023 arXiv

  51. [60]

    Prompting gpt-3 to be reliable

    Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Boyd-Graber, and Lijuan Wang. Prompting gpt-3 to be reliable. arXiv preprint arXiv:2210.09150, 2022

    2022 arXiv

  52. [61]

    The good, the bad, and the greedy: Evaluation of llms should not ignore non-determinism

    Yifan Song, Guoyin Wang, Sujian Li, and Bill Yuchen Lin. The good, the bad, and the greedy: Evaluation of llms should not ignore non-determinism. arXiv preprint arXiv:2407.10457, 2024

    2024 arXiv

  53. [62]

    Analysis of variance (anova).Chemometrics and intelligent laboratory systems, 6(4):259–272, 1989

    Lars St, Svante Wold, et al. Analysis of variance (anova).Chemometrics and intelligent laboratory systems, 6(4):259–272, 1989

    1989

  54. [63]

    One embedder, any task: Instruction-finetuned text embeddings

    Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A Smith, Luke Zettlemoyer, and Tao Yu. One embedder, any task: Instruction-finetuned text embeddings. arXiv preprint arXiv:2212.09741, 2022

    2022 arXiv

  55. [64]

    Grok 3 appears to have briefly censored unflattering mentions of trump and musk, 2025

    TechCrunch. Grok 3 appears to have briefly censored unflattering mentions of trump and musk, 2025. URL https://techcrunch.com/2025/02/23/ grok-3-appears-to-have-briefly-censored-unflattering-mentions-of-trump-and-musk/ . Accessed: 2025-05-03

    2025

  56. [65]

    A robust bias mitigation procedure based on the stereotype content model

    Eddie L Ungless, Amy Rafferty, Hrichika Nag, and Björn Ross. A robust bias mitigation procedure based on the stereotype content model. arXiv preprint arXiv:2210.14552, 2022

    2022 arXiv

  57. [66]

    Llm leaderboard, 2025

    Vellum AI. Llm leaderboard, 2025. URL https://www.vellum.ai/llm-leaderboard. Accessed: 2025-05-12

    2025

  58. [67]

    The dark side of generative artificial intelligence: A critical analysis of controversies and risks of chatgpt

    Krzysztof Wach, Cong Doanh Duong, Joanna Ejdys, R ¯uta Kazlauskait ˙e, Pawel Korzynski, Grzegorz Mazurek, Joanna Paliszkiewicz, and Ewa Ziemba. The dark side of generative artificial intelligence: A critical analysis of controversies and risks of chatgpt. Entrepreneurial Busin...

    2023

  59. [68]

    Large language models are not fair evaluators

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926, 2023

    2023 arXiv

  60. [69]

    Self-instruct: Aligning language models with self-generated instructions

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560, 2022

    2022 arXiv

  61. [70]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Pr...

    2024

  62. [71]

    All of statistics: a concise course in statistical inference

    Larry Wasserman. All of statistics: a concise course in statistical inference. Springer Science & Business Media, 2013

    2013

  63. [72]

    Measuring and reducing gendered correlations in pre-trained models

    Kellie Webster, Xuezhi Wang, Ian Tenney, Alex Beutel, Emily Pitler, Ellie Pavlick, Jilin Chen, Ed Chi, and Slav Petrov. Measuring and reducing gendered correlations in pre-trained models. arXiv preprint arXiv:2010.06032, 2020

    2010 arXiv

  64. [73]

    The generalization of ‘student’s’problem when several different population varlances are involved

    Bernard L Welch. The generalization of ‘student’s’problem when several different population varlances are involved. Biometrika, 34(1-2):28–35, 1947

    1947

  65. [74]

    Livebench: A challenging, contamination-free LLM benchmark

    Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz- Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh-Agrawal, Sandeep Singh Sandha, Siddartha Venkat Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and ...

    2025

  66. [75]

    This powerful new chatbot works great—unless you ask about china, 2025

    Wired. This powerful new chatbot works great—unless you ask about china, 2025. URL https: //www.wired.com/story/deepseek-censorship/. Accessed: 2025-05-03

    2025

  67. [76]

    Compensatory debiasing for gender imbalances in language models

    Tae-Jin Woo, Woo-Jeoung Nam, Yeong-Joon Ju, and Seong-Whan Lee. Compensatory debiasing for gender imbalances in language models. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023

    2023

  68. [77]

    Order matters in hallucination: Reasoning order as benchmark and reflexive prompting for large-language-models

    Zikai Xie. Order matters in hallucination: Reasoning order as benchmark and reflexive prompting for large-language-models. arXiv preprint arXiv:2408.05093, 2024

    2024 arXiv

  69. [78]

    Large language model as attributed training data generator: A tale of diversity and bias

    Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander J Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. Large language model as attributed training data generator: A tale of diversity and bias. Advances in Neural Information Processing Systems, 36:55734–55784, 2023

    2023

  70. [79]

    Wider and deeper llm networks are fairer llm evaluators

    Xinghua Zhang, Bowen Yu, Haiyang Yu, Yangyu Lv, Tingwen Liu, Fei Huang, Hongbo Xu, and Yongbin Li. Wider and deeper llm networks are fairer llm evaluators. arXiv preprint arXiv:2308.01862, 2023

    2023 arXiv

  71. [80]

    Explainability for large language models: A survey

    Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. Explainability for large language models: A survey. ACM Transactions on Intelligent Systems and Technology, 15(2):1–38, 2024

    2024

  72. [81]

    Gptbias: A comprehensive framework for evaluating bias in large language models

    Jiaxu Zhao, Meng Fang, Shirui Pan, Wenpeng Yin, and Mykola Pechenizkiy. Gptbias: A comprehensive framework for evaluating bias in large language models. arXiv preprint arXiv:2312.06315, 2023

    2023 arXiv

  73. [82]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    2023

  74. [83]

    Lima: Less is more for alignment

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36:55006–55021, 2023

    2023

  75. [84]

    {question}

    Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations, 2022. 15 A Technical Appendices and Supplementa...

    2022

  76. [85]

    A.3 Bias Criteria Table Score Description 1 Fully neutral and factual

    and LLM-as-a-judge ones with Gemini (Figure 2) and GPT-4o are the same. A.3 Bias Criteria Table Score Description 1 Fully neutral and factual. Provides objective information with no emotive tone or ideological framing. Acknowledges all perspectives equally. 2 Mostly balanced a...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.