Pith. sign in

REVIEW 4 major objections 6 minor 2 references

Most measured LLM self-preference is evaluator noise, not narcissism, this paper argues.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-03 06:30 UTC pith:UE47YYAL

load-bearing objection The outcome-matched proxy control is a real step forward, but the 89.6%/10.4% headline overstates what the design can support. the 4 major comments →

arxiv 2601.22548 v4 pith:UE47YYAL submitted 2026-01-30 cs.CL cs.AIcs.LG

Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations

Dani Roytburg , Matthew Bozoukov , Matthew Nguyen , Jou Barzdukas , Mackenzie Puig-Hall , Narmeen Oozeer This is my paper
classification cs.CL cs.AIcs.LG
keywords LLM evaluatorsself-preference biasevaluator uncertaintyjudge biasoracle labelsevaluation quality baselinechain-of-thoughtmodel ranking
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to test whether large language models really favor their own outputs when acting as judges, or whether that appearance is a statistical artifact. It introduces an Evaluator Quality Baseline: on questions a judge answers incorrectly, the judge's voting behavior on its own wrong answer is compared with its voting behavior on a capability-matched wrong answer from a different model. Across 37,448 evaluation pairs from nine datasets and sixteen models, roughly 89.6% of measured self-preference disappears against this control, and only about half of previously significant findings survive. The authors conclude that most 'narcissism' signals are driven by evaluator uncertainty on hard items, while a smaller residual bias remains, concentrated in cases like MMLU. A careful reader would care because automated evaluation pipelines that rely on LLM judges may be mis-calibrated — and mitigation efforts may target the wrong models — if self-preference is not measured against an uncertainty baseline.

Core claim

The paper's central claim is that previously reported self-preference bias in LLM evaluators is dominated by an experimental confound: when a judge model is asked to compare answers to a question it itself got wrong, it tends to vote for an incorrect response at random, and that random preference has been misread as favoritism toward its own output. To separate the two, the authors define an Evaluator Quality Baseline that pairs each incorrect self-evaluation with the judge's evaluation of an equally incorrect response from a different model, so no 'self' is present. The difference between those two vote probabilities is the true self-preference signal. Measured this way, self-preference dro

What carries the argument

The Evaluator Quality Baseline (EQB) is the paper's central device. The judge's preference probability for its own response is compared, on the same set of 'hard' examples (where the judge produced an incorrect or inferior answer), with its preference probability for a proxy response from another model that received the same oracle label. The difference, averaged and tested with a paired t-test, isolates the causal effect of the response being one's own rather than the effect of evaluating a poor answer in general. The machinery also includes a decomposition of bias into 'illegitimate' (unearned credit on wrong answers) and 'legitimate' (recognized merit on right answers) components, showing

Load-bearing premise

The ground-truth labels that mark responses as correct or incorrect must be unbiased; for subjective datasets they are majority votes from other LLM judges, so any shared stylistic bias in those oracles would make the baseline subtract genuine differences instead of uncertainty.

What would settle it

Re-run the procedure on a subjective dataset with human gold labels instead of LLM-oracle labels; if the residual self-preference grows or the baseline reduction shrinks materially, the oracle assumption is what carries the result. A cheaper check: on an objective dataset like MATH500, verify on a human-annotated subset that the proxy's incorrect responses are truly equivalent in quality to the judge's incorrect responses, not systematically worse or better.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X LinkedIn Reddit HN

If this is right

  • Across the reproduced studies, evaluator uncertainty accounts for an average of 89.6% of measured self-preference; only 10.4% exceeds a control-group baseline.
  • Only about half of previously significant self-preference findings remain statistically significant after applying the baseline, and 44% of experiments show no or negative bias.
  • Model rankings by self-preference shift substantially: for example, Llama 3.1-8B on AlpacaEval falls from the highest initial bias to near the lowest, while Qwen2.5-72B on code moves to the highest rank.
  • Chain-of-thought prompting does not reliably reduce self-preference once the baseline is applied; its effect varies by model and dataset.
  • A genuine residual self-preference signal persists on MMLU, especially for the Qwen2.5 family, indicating that some real bias remains beyond the uncertainty artifact.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the oracle labels used for subjective tasks carry any shared stylistic or quality bias, the baseline could subtract genuine self-preference along with uncertainty; the reported reduction should therefore be read as an upper bound on the noise artifact until human-label replications are run.
  • The entropy result — that vote confidence is unchanged whether a judge rates itself or a proxy — offers a testable signature: genuine self-recognition might be expected to change confidence on self-text, so its absence supports a noise-based rather than identity-based mechanism.
  • Because the corrected measurements re-rank models, mitigation strategies such as steering vectors built on uncorrected self-preference scores may have targeted the wrong models; re-running those interventions on EQB-adjusted rankings is a direct next step.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes an Evaluator Quality Baseline to separate genuine self-preference bias in LLM judges from evaluator uncertainty on examples where the judge's own response is incorrect. On a reproduction of four prior self-preference pipelines across 37,448 comparison pairs, the authors report that capability-matched proxy responses eliminate, on average, 89.6% of measured self-preference, that only 51% of original findings remain statistically significant against the new null, and that only 10.4% of measured self-preference exceeds the control baseline. The paper also reports entropy analyses suggesting that judge uncertainty on hard examples is similar for self and proxy evaluations, and concludes that prior model rankings by self-preference are substantially affected by evaluator-quality artifacts.

Significance. If the central claim holds, this is a valuable methodological correction to the growing LLM-as-judge literature: it provides a concrete baseline for separating self-preference from evaluator noise on hard examples, and the Eq. (5) decomposition of bias into legitimate and illegitimate components is correct and useful. The paper's strength is its large-scale reproduction of four landmark self-preference studies with example-level outcome matching, proxy validation, and sensitivity checks. The notion that much reported self-preference may be an artifact of evaluator uncertainty is plausible and important. However, the headline quantitative claims currently rest on an assumption of exchangeability within binary outcome classes, which is not established, and on summary statistics that are not precisely reconciled.

major comments (4)
  1. [§3.3, Eqs. (6)–(8), and §5.1.1] The null hypothesis H0: T_quality ≤ 0 assumes that, absent self-preference, the judge's vote distribution is identical for its own incorrect response and a proxy's incorrect response. This exchangeability is not guaranteed by the oracle indicator G, which is binary. Two responses can both be 'incorrect' and yet differ substantially in partial correctness, informativeness, or proximity to the reference; a judge with even weak quality sensitivity will then prefer its own answer to a worse proxy answer (or vice versa) without any self-recognition, making Δs_J nonzero under the null. The paper concedes this in §5.1.1 ('does not guarantee that the proxy could not be substantially better or worse than the judge on the same example'), and the model-level R²=79% in Fig. 4 does not rule out per-example mismatches. This threatens the specific interpretation of the 89.6% reduction and the 10.4% res
  2. [§1, §7, Table 1] The headline statistics are not mutually reconciled and are reported without uncertainty. The abstract states that only 51% of initial findings retain statistical significance, the introduction states that 50% report values that lose significance, and the conclusion that only 10.4% of self-preference bias exceeds the baseline; these are not obviously consistent, and the '44% of experiments should show no or negative bias' figure adds a third counting rule. The 89.6% figure is an unweighted average of per-experiment relative reductions, many of which exceed 100% (e.g., Table 1 rows with Rel.∆ ≈ −120% or −167%). Please report absolute reductions with standard errors/confidence intervals, specify whether the average is weighted by N or by dataset, and define precisely the population over which each percentage (51%, 50%, 44%, 10.4%) is computed.
  3. [§4.2.2, §6.1] For all subjective tasks, the oracle labels G used to define the ILSP subset and to match proxies are majority votes of 'neutral' LLM judges (GPT-4o-mini, Gemini-1.5-Flash, DeepSeek-V3, and GPT-5.2). If these oracles share stylistic preferences, position bias, or any family-level self-preference, examples can be mislabeled as 'incorrect' and proxy outcomes no longer match true quality. The authors explicitly acknowledge that these labels 'should not be interpreted as definitive gold standards,' but the central quantitative claims on AlpacaEval, TruthfulQA, WMT19, CNN, and XSUM depend on this oracle. Please provide a sensitivity analysis using a human-labeled subset (the human validation in §4.2.3 is a start) or report oracle–human agreement as a covariate, and show how the 89.6% reduction and the 10.4% residual change under alternative oracle definitions.
  4. [Table 1 and §4.1] The significance claims are based on raw p-values from a paired t-test over many experimental configurations, with no multiple-testing correction. Some cells have small N (e.g., MATH500 N=20–144), and the count of 'significant' experiments is likely unstable. Report confidence intervals for the per-configuration differences, apply a multiple-comparison procedure (e.g., FDR control) when counting how many findings 'retain significance,' and avoid relying solely on p<0.05 dichotomies.
minor comments (6)
  1. [Abstract and §1] Please make the headline statistics consistent across the abstract, introduction, and conclusion, and add confidence intervals for the 89.6% average reduction.
  2. [§3.2] Typo: 'Chen et al. [2025a] refer to this metric is referred to as ...' should read '... refer to this metric as ...'.
  3. [Eq. (5)] The decomposition is correct, but the sentence 'self-preference bias (unearned credit) is exclusively a function of ILSP' is overstated: the overall Bias also depends on Acc and LSP through the weighting. Clarify that ILSP is the component of interest.
  4. [Table 1] The formula for Rel.∆ has a typographical issue (missing parentheses around ILSP_orig), and the note should explain why reductions can exceed 100%.
  5. [§5.3 and Fig. 5] The text reports both R²=73% and ρ=0.85 for the same entropy relation; state clearly which is which, and note whether the correlation is on model-level aggregates or per-example values.
  6. [Appendix B.4] The claim that 'a substantial fraction of the dataset retains multiple proxies per example' would benefit from specific percentages; the figures alone are hard to read.

Circularity Check

0 steps flagged

No significant circularity: the Evaluator Quality Baseline is an empirical control and the 89.6% figure is data-dependent, not a consequence of the definitions.

full rationale

The derivation chain is: oracle labels G define ILSP (Eq. 5); proxies are selected on outcome matching G(x,o_J,o_R)=G(x,o_K,o_R) (Eq. 6); the test statistic is the mean difference s_J(x,o_J,o_R)-s_J(x,o_K,o_R) over that subset (Eqs. 7-9). The headline quantities (51% retention, 89.6% average reduction, 10.4% residual) are computed from observed vote probabilities and vary across models/datasets (Table 1), so they are not forced by the equations: nothing in the setup makes the proxy-vote rate equal 89.6% of the self-vote rate a priori. The H0: T_quality<=0 test is a substantive empirical null, and Fig. 4/App. B provide independent validation of the matching. The two most salient self-referential risks are explicitly conceded rather than hidden: §5.1.1 says the oracle signal 'does not guarantee that the proxy could not be substantially better or worse than the judge on the same example,' and §6.1 says model-derived oracle labels for subjective tasks 'may encode other biases unrelated to authorship.' These are validity/attribution assumptions about exchangeability under the null, not circular reductions of the result to its inputs; they could change the interpretation of the residual, but they do not render the derivation equivalent to its presuppositions. The only self-citation (Roytburg et al. 2025 in §5.1, for steering-vector mitigation as an application) is incidental and not load-bearing for the baseline or the headline statistics. No step of the claimed derivation is self-definitional, fitted-then-predicted, or dependent on a self-cited uniqueness/ansatz chain.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

No free parameters are fitted; the 89.6% is a data-derived average reduction, not a model parameter. The main assumed inputs are the oracle labels, the outcome-matching proxy assumption, and the paired t-test setup.

axioms (4)
  • standard math The paired t-test is valid: ILSP differences across examples are independent and identically distributed.
    Used for T_quality in §4.1; no clustering or dependence correction is described.
  • domain assumption Oracle labels G from 'neutral' LLM judges are unbiased for subjective tasks.
    Load-bearing for defining ILSP and matching proxies; acknowledged as a limitation in §6.1.
  • domain assumption Example-level outcome matching (G(x,o_J,o_R)=G(x,o_K,o_R)) ensures proxy responses are exchangeable with the judge's own response in non-self aspects.
    This is the core of the Evaluator Quality Baseline (§3.3, Eq. 6); if proxies differ in style or non-outcome quality, the baseline may over-correct.
  • ad hoc to paper The one-sided null H0: T_quality ≤ 0 is the correct hypothesis for self-preference.
    Chosen in §3.3; the paper does not justify why negative differences (judge prefers proxy) are treated as absence of bias rather than a separate phenomenon.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations." pith.science (2026). https://pith.science/paper/UE47YYAL

@misc{pith2026260122548,
  author       = {Pith},
  title        = {Pith review of: Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UE47YYAL}},
  note         = {Machine review of arXiv:2601.22548}
}
Share X LinkedIn Reddit HN
read the original abstract

Recent research has shown that large language models (LLMs) favor their own outputs when acting as judges, undermining the integrity of automated post-training and evaluation workflows. However, it is difficult to disentangle which behaviors are explained by narcissism versus experimental confounds. Specifically, LLM evaluators may deliver self-preferring verdicts when comparing responses to questions they fail on; these verdicts may not depend on the identity of the author, but on evaluator quality. We correct this by directly comparing the judge's voting distribution in cases where it evaluates itself versus another model. This evaluator quality baseline reveals that only 51% of examples in previous findings retain statistical significance against this null hypothesis, covering 89.6% of total self-preference probability mass. Finally, we compare the entropy of voting distributions, suggesting uncertainty-driven overlap, and show that our procedure enables more careful documentation against the backdrop of judge-bias research.

Figures

Figures reproduced from arXiv: 2601.22548 by Dani Roytburg, Jou Barzdukas, Mackenzie Puig-Hall, Matthew Bozoukov, Matthew Nguyen, Narmeen Oozeer.

Figure 2
Figure 2. Figure 2: Judge Task Accuracy versus Illegitimate Self-Preference (Sec. 4.2.1). With our proposed [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: More results on Judge Task Accuracy versus original (light) and updated (full) self [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Model-level winrate of judges versus weighted average winrate of selected proxies. Each [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Shannon Entropy on hard (ILSP) example distributions is strongly correlated, regardless of [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Sensitivity of Evaluator Quality Baseline to inclusion of same-family proxies. Each point [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Mean test statistic versus number of proxies per example. [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Mean test statistic versus number of proxies per example. [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Mean test statistic versus number of proxies per example. [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Mean test statistic versus number of proxies per example. [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Percentage of examples with at least N valid proxies. [PITH_FULL_IMAGE:figures/full_fig_p020_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Percentage of examples with at least N valid proxies. [PITH_FULL_IMAGE:figures/full_fig_p020_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Percentage of examples with at least N valid proxies. [PITH_FULL_IMAGE:figures/full_fig_p020_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Percentage of examples with at least N valid proxies. [PITH_FULL_IMAGE:figures/full_fig_p021_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Shannon Entropy Gap (Eq. 11) versus illegitimate self-preference. 33 of 49 experiments [PITH_FULL_IMAGE:figures/full_fig_p026_15.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

2 extracted references · 2 linked inside Pith

  1. [2017]

    doi: 10.18653/v1/P17-1099

    Association for Computational Linguistics. doi: 10.18653/v1/P17-1099. URL https: //www.aclweb.org/anthology/P17-1099. Chenhui Shen, Liying Cheng, Xuan-Phi Nguyen, Yang You, and Lidong Bing. Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Findings of the Associati...

    Pith/arXiv arXiv 2023

  2. [2025]

    family members

    URLhttp://arxiv.org/abs/2410.21819. Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V . Chawla, and Xiangliang Zhang. Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge. InThe Thirteenth International Conference on Learning Representations, October 2024. URLhttps://op...

    Pith/arXiv arXiv 2024

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.