Pith. sign in

REVIEW 5 major objections 6 minor 3 cited by

Breaking the Mirror: Activation-Based Mitigation of Self-Preference in LLM Evaluators

T0 review · 5 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper shows that a single additive steering vector, applied to an LLM judge's internal activations at inference time, flips up to 97 percent of unjustified self-preferring choices on XSUM summarization pairs, while leaving the model's

desk verdict The idea is worth pursuing, but the 97% flip rate is not yet probative because the vectors are evaluated on the same examples they are built from. read the letter →

arxiv 2509.03647 v2 pith:EAPI4YYQ submitted 2025-09-03 cs.CL cs.AIcs.LG

Dani Roytburg , Matthew Bozoukov , Matthew Nguyen , Jou Barzdukas , Simon Fu , Narmeen Oozeer This is my paper
classification cs.CLcs.AIcs.LG
keywords self-preferencebiasLLM-as-a-judgesteeringvectorscontrastiveactivationadditionengineeringdirectpreferenceoptimizationXSUMevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LLMs used as judges tend to prefer their own outputs, which corrupts preference tuning and model routing. This paper asks whether a lightweight steering vector, added to the judge's internal activations at inference time, can suppress that bias without retraining. Using a curated XSUM dataset that separates decisions into unjustified self-preference, justified self-preference, and unbiased agreement, the authors show that both contrastive activation addition and a gradient-based vector flip up to 97 percent of unjustified self-preferring choices—far beyond prompting (0 percent) or direct preference optimization (49 percent). The same vectors also disrupt many justified self-preferring choices and unbiased agreements, which the authors read as evidence that self-preference is not captured by one linear direction. If correct, the work offers a cheap, targeted intervention for LLM-as-a-judge pipelines while clarifying the limits of linear steering.

What carries the argument

The key mechanism is the steering vector: an additive vector added to the residual-stream activations of the judge model at a chosen layer while it scores pairs. Two constructions are tested. CAA (contrastive activation addition) takes the mean hidden-state difference between biased and unbiased completions and uses that as a steering direction; the optimization approach learns an additive vector by gradient descent on a contrastive log-loss that promotes the desired completion and suppresses the biased one. The other load-bearing piece is the gold-label majority vote: three LLM judges from different model families (Phi-4, DeepSeek V3, Claude 3.5-Sonnet) decide, per pair, which summary is ac

What would settle it

Human-annotate a random subset of the XSUM pairs and re-label each judge decision as justified or biased; if the steering vector flips about 97 percent of human-labeled biased decisions but also flips a large share of human-labeled correct decisions, the claim of bias-specific mitigation fails. A second check: apply a random steering vector of the same magnitude; comparable flip rates would indicate a generic choice shift rather than a self-preference-specific direction.

Watch

Extended reading notes

Core claim

The paper's central claim is that self-preference bias in LLM evaluators has a substantial linear component in activation space, and that this component can be cancelled by an additive steering vector at inference time. On XSUM summarization pairs, with Llama-3.1-8B-Instruct as the judge and GPT-3.5 as the comparison model, the authors separate each self-choice into illegitimate self-preference (judge prefers itself while gold judges prefer the other), legitimate self-preference (both prefer the judge), and unbiased agreement (both prefer the other). CAA and an optimization-based steering vector each flip 97 percent of illegitimate self-preferences, versus 0 percent for an explicit prompt wa

Load-bearing premise

The boundary between unjustified self-preference and justified self-preference is set by the majority vote of three LLM judges, not by human ground truth; if those judges share a bias, the 97 percent reduction measures alignment with the gold panel rather than removal of real bias.

Editorial extensions

If this is right

  • LLM-as-a-judge pipelines can apply a one-vector activation edit at inference time to remove most unjustified self-preference without retraining, at near-zero marginal cost.
  • Optimization-based steering matches CAA's 97 percent flip rate with far fewer examples, making it practical when labeled biased examples are scarce.
  • Because prompting alone changes no biased decisions, activation editing is a qualitatively stronger mitigation than instruction-style reminders.
  • The vectors' instability on legitimate self-preference and unbiased agreement means a deployed safeguard would need to be selective or combined with another method.
  • The strong result in both aware and unaware settings implies the bias signal enters the residual stream even when authorship is hidden, so mitigation does not depend on detecting an explicit 'mine' cue.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural control the paper does not run: compare the steering vector against a random or opposite-direction vector of the same magnitude; if the random vector also flips about 97 percent of biased examples, the effect is a generic choice shift rather than a self-preference-specific correction.
  • Because the gold judges are themselves LLMs, the 'illegitimate' label is a model-consensus construct. Human annotation on a subsample would test whether the 97 percent flip rate corresponds to human-judged bias or simply realigns the judge with the gold panel's shared preferences.
  • The instability on legitimate cases suggests a two-vector or conditional steering scheme, one direction for bias suppression and another for preserving correct self-preference, which could be tested directly on the same dataset.
  • Since the dataset is XSUM-only, a natural extension is to repeat the same split-and-steer protocol on other pairwise tasks, such as coding or instruction following, to see whether the linear-bias finding generalizes beyond summarization.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper studies self-preference bias in LLM-as-judge settings and proposes to mitigate it with activation steering vectors. On an XSUM pairwise summarization benchmark, the authors label each judge decision as illegitimate self-preference, legitimate self-preference, or unbiased agreement using a majority vote of three gold LLM judges. They construct steering vectors via Contrastive Activation Addition (CAA) and a gradient-based optimization method, and measure flip rates on the three subsets. Table 1 reports that three of four steering conditions flip 97% of illegitimate self-preference decisions, outperforming a prompting baseline (0%) and a DPO baseline (49%). However, the same vectors flip a large fraction of legitimate self-preference and unbiased agreement decisions, which the authors interpret as evidence that self-preference is not a single linear direction. Code is released.

Significance. If the 97% flip rate is a valid held-out estimate, the paper would show a remarkably cheap and effective inference-time intervention for a known bias in LLM evaluation pipelines. The curated XSUM split and the comparison against prompting and DPO are useful resources for the community. The main weakness is internal validity: the optimization vector is trained on the same biased examples whose flip rate is later reported in Table 1, and no held-out split, confidence intervals, or human validation of the gold labels are provided. Until these issues are addressed, the headline quantitative claim cannot be considered established.

major comments (5)
  1. [§2.2.2 / Appendix B / Table 1] The optimization-based steering vector is trained on 'the top 20th percentile of prompts... given that the gold label votes for model K' (Appendix B), which is exactly the Bias subset reported in Table 1. No train/test split is described, and §2.2.2 states layers were selected 'as those performed the best on our evaluations.' Thus the 97% flip rate for Optimization is potentially an in-sample training accuracy, not an unbiased estimate of mitigation effectiveness. A held-out evaluation (e.g., 5-fold CV or a construction/evaluation split) is required, and the sizes of the three subsets after discarding position-biased items must be reported.
  2. [§2.1 / §2.2.1] The CAA vector is built from 'prompts p+ that yield unbiased completions and prompts p− which yield biased completions' without stating that the evaluation set is disjoint from this construction set. If the same biased/unbiased examples are used both to build vCAA and to compute flip rates in Table 1, the 97% CAA result has the same circularity problem as the optimization method. The paper must explicitly disclose the split (or lack thereof) and, if there is no split, rerun the evaluation on held-out examples.
  3. [§2.1 / §3] The 'ground-truth' gold label gi is the majority vote of three LLM judges (Phi-4, DeepSeek V3, Claude 3.5-Sonnet). No human validation, per-judge agreement statistics, or sensitivity analysis is reported. If the gold panel itself exhibits self-preference or other biases, the very definition of 'illegitimate self-preference'—and hence the meaning of the 97% flip rate—is undermined. At minimum, report agreement among the three gold judges and validate a random sample against human annotations.
  4. [Table 1 / §3] The stability columns show that the 97%-flip vectors also flip 47–93% of legitimate self-preference (LSP) and 20–50% of unbiased agreement. This pattern suggests a global 'choose the other response' shift rather than a targeted correction of illegitimate self-preference. The abstract's claim that steering 'reduce[s] unjustified self-preference bias' is therefore too strong: the intervention trades away a large fraction of correct decisions. The authors should report a combined utility metric (e.g., net accuracy on all subsets) and temper the conclusion accordingly.
  5. [Table 1 / §3] No confidence intervals, standard errors, or significance tests are provided, and the flip rates appear to come from a single run. Since the subsets are formed after discarding position-biased items, the effective sample sizes are unknown and could be small. The 'up to 97%' headline is not accompanied by any uncertainty estimate; bootstrap or multi-seed repetition is needed to establish that the effect is not within noise.
minor comments (6)
  1. [Abstract / Table 1] The abstract states that 'steering vectors can reduce unjustified self-preference bias by up to 97%,' but only three of four steering conditions in Table 1 reach 97%; the aware Optimization condition is 23%. Qualify the claim with the specific setting.
  2. [Figure 1 caption] Typos: 'aligned meanin blind' and 'aligned mean' should read 'aligned mean in blind' and 'aligned mean'. Also, the caption does not define ' µ judge' or how the curves are computed.
  3. [§2.2.1] It is unclear what 'unbiased completions' and 'biased completions' mean concretely. Do they refer to J's choice matching the gold label, or to the gold label itself? Define these terms precisely.
  4. [Table 1 / Appendix D] The DPO baseline is described as trained in the unaware setting (Appendix D), but Table 1 does not specify whether DPO results are for the aware or unaware evaluation. Add a row/column label or note.
  5. [Appendix F] The sample outputs contain many encoding artifacts (e.g., 'över', '¨', 'Äll', 'or ' and '1ör'). These should be cleaned before publication; they currently obscure the qualitative examples.
  6. [References] There are minor reference formatting issues, such as duplicated initials in the DeepSeek-V3 author list and inconsistent URL access styles. A final pass with a consistent bibliography style is recommended.

Circularity Check

2 steps flagged · score 6.0 of 10

The headline 97% reduction is at least partly a training-set score: steering vectors are constructed from the same examples used to measure effectiveness, with no held-out split reported.

  1. fitted input called prediction [Appendix B; Table 1]
    "For our experiments, we froze the model weights, hooked an additive vector, and optimized over the top 20th percentile of prompts based on the highest probability that J would select its own answer, given that the gold label votes for model K, maximizing the probability of target completion Y+ corresponding to model K's response and minimizing the initial, biased completion Y− = vJ."

    This optimization is performed on examples that satisfy exactly the paper's definition of the 'Bias' subset: J selects its own summary while the gold judges prefer K's summary. Table 1 then reports the flip rate 'computed within' the Bias subset, and no train/test split is stated anywhere in the paper. The optimization loss in Eq. (1), -log P(Y+|X;h) - log(1-P(Y-|X;h)), is precisely the objective of flipping those biased choices. Therefore, on the fitted examples, the reported 97% effectiveness is the training objective itself, not an independent prediction. The abstract's 'up to 97%' claim is thus at least partly in-sample.

  2. fitted input called prediction [Section 2.2.1; Table 1]
    "Formally, given a dataset X of prompts p paired with completions c generated by model J with greedy sampling, we select prompts p+ that yield unbiased completions c+ and prompts p− which yield biased completions c−, we then define the CAA vector vCAA for a model layer L as follows:"

    The CAA vector is built by averaging activations on 'biased' (p−) and 'unbiased' (p+) examples, which are drawn from the same subsets used to compute flip rates in Table 1 (Bias and Agreement). No held-out split is reported. Consequently, the 0.97 effectiveness on the Bias subset is a training-set score: the direction is a linear function of the activations on the very examples whose flip rate is being reported, so it does not demonstrate held-out generalization. This makes the central 'up to 97%' result at least partially a fit to the evaluation set rather than a prediction.

full rationale

The paper's central claim is that steering vectors 'reduce unjustified self-preference bias by up to 97%'. The evidence for this is Table 1, which reports flip rates on the Bias subset. However, the optimization-based vector is explicitly fit to the top 20th percentile of prompts satisfying the Bias definition, and the CAA vector is constructed from biased/unbiased examples in the same dataset. The paper never states that the evaluation examples are disjoint from the training/construction examples. On the fitted portion, the measured flip rate is the training objective (for optimization) or a direct average of the same examples (for CAA), so the headline number is at least partly in-sample. This is a genuine internal-validity problem: the fitted input is being called a prediction. The gold-judge majority-vote definition of legitimacy is an external assumption, not circular, because the gold judges are different models from the judge being steered. There are no load-bearing self-citations. The instability results (high LSP flips, low Agreement flips) are separate correctness concerns about the intervention's selectivity, not additional circularity. Overall, the paper's main quantitative result is partially forced by construction, warranting a score of 6.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim relies on a curated dataset of self-preference examples labeled by three LLM judges, a steering multiplier, layer selection, and a training-percentile threshold. No new physical or conceptual entities are introduced.

free parameters (3)
  • steering multiplier = 0.1 for Table 1; 0.5 for CAA in Fig. 1
    Chosen to balance effectiveness and stability; likely selected on the evaluation set.
  • layer index = 14 for optimization; 14-16 for CAA
    Layer chosen as best on evaluations, reported in Section 2.2.2 and Appendix B.
  • top-20th-percentile threshold = 20%
    Prompts with the highest probability of self-selection when gold votes for K are used to train the optimization vector; threshold is ad hoc.
assumptions (3)
  • domain assumption LLM gold judges (Phi-4, DeepSeek V3, Claude 3.5-Sonnet) provide reliable ground truth for summary quality
    The split into illegitimate/legitimate self-preference and unbiased agreement depends on the majority vote of these models; no human evaluation is provided (Section 2.1).
  • domain assumption Self-preference bias can be represented by a linear direction in activation space
    The method assumes a single additive vector can modulate bias; the instability results call this into question (Section 5) but the method still relies on it.
  • domain assumption Activations at the last 10 token positions and specific layers capture the relevant behavior
    Layer 14 was selected by performance; this is an empirical choice, not a derived one.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Breaking the Mirror: Activation-Based Mitigation of Self-Preference in LLM Evaluators." pith.science (2026). https://pith.science/paper/EAPI4YYQ

@misc{pith2026250903647,
  author       = {Pith},
  title        = {Pith review of: Breaking the Mirror: Activation-Based Mitigation of Self-Preference in LLM Evaluators},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EAPI4YYQ}},
  note         = {Machine review of arXiv:2509.03647}
}
read the original abstract

Large language models (LLMs) increasingly serve as automated evaluators, yet they suffer from "self-preference bias": a tendency to favor their own outputs over those of other models. This bias undermines fairness and reliability in evaluation pipelines, particularly for tasks like preference tuning and model routing. We investigate whether lightweight steering vectors can mitigate this problem at inference time without retraining. We introduce a curated dataset that distinguishes self-preference bias into justified examples of self-preference and unjustified examples of self-preference, and we construct steering vectors using two methods: Contrastive Activation Addition (CAA) and an optimization-based approach. Our results show that steering vectors can reduce unjustified self-preference bias by up to 97\%, substantially outperforming prompting and direct preference optimization baselines. Yet steering vectors are unstable on legitimate self-preference and unbiased agreement, implying self-preference spans multiple or nonlinear directions. This underscores both their promise and limits as safeguards for LLM-as-judges and motivates more robust interventions.

Figures

Figures reproduced from arXiv: 2509.03647 by the authors.

Figure 1
Figure 1. A steering vector fits a self-preferring model around an aligned mean in blind (left) and aware (right) pairwise preference tests, suggesting the representation of self-preference can be derived from linear space. Steering on layer 14 with a multiplier of 0.5 (CAA) and 0.1 (Optimization). We mitigate this with steering vectors—lightweight, inference-time activation edits with minimal training cost [Im and Li, 2025].… view at source ↗
Figure 2
Figure 2. Probability of the self-evaluating model [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Probability of the self-evaluating model [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Probability of the self-evaluating model [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Plot of the distribution of a model’s probability of selecting its own output on the APPS [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Self-Preference Bias in Rubric-Based Evaluation of Large Language Models

    cs.CL 2026-04 unverdicted novelty 7.0 of 10

    Rubric-based LLM judges show self-preference bias, incorrectly marking their own failed outputs as satisfied up to 50% more often on verifiable benchmarks and skewing scores by 10 points on subjective ones.

  2. Self-Preference Bias in Rubric-Based Evaluation of Large Language Models

    cs.CL 2026-04 conditional novelty 7.0 of 10

    LLM judges over-approve their own outputs in rubric-based evaluation, even when rubrics are programmatically verifiable, and ensembling only partially corrects the bias.

  3. Self-Preference Bias in Rubric-Based Evaluation of Large Language Models

    cs.CL 2026-04 conditional novelty 6.0 of 10

    Self-preference bias persists in rubric-based LLM evaluation even with fully objective, programmatically verifiable rubrics, and can shift subjective medical-chat scores by up to ~10 points.

Reference graph

Works this paper leans on

17 extracted references · 16 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Phi-4 technical report

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905,

    arXiv

  2. [5]

    Mingzhe Du, Anh Tuan Luu, Xiaobao Wu, Yichong Huang, Yue Liu, Dong Huang, Huijun Liu, Bin Ji, Jie M

    URL https://arxiv.org/abs/2412.19437. Mingzhe Du, Anh Tuan Luu, Xiaobao Wu, Yichong Huang, Yue Liu, Dong Huang, Huijun Liu, Bin Ji, Jie M. Zhang, and See-Kiong Ng. Collective Bias Mitigation via Model Routing and Collaboration. June

    arXiv

  3. [6]

    arXiv:2502.18862 [cs]

    URL http://arxiv.org/abs/2502.18862. arXiv:2502.18862 [cs]. Víctor Gallego. Configurable preference tuning with rubric-guided synthetic data. arXiv preprint arXiv:2506.11702,

    arXiv

  4. [7]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  5. [8]

    arXiv:2411.15594 [cs]

    URL http://arxiv.org/abs/ 2411.15594. arXiv:2411.15594 [cs]. Shawn Im and Yixuan Li. A unified understanding and evaluation of steering methods.arXiv preprint arXiv:2502.02716,

    arXiv

  6. [9]

    Shashi Narayan, Shay B

    URL https://arxiv.org/abs/2311.09766. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization,

    arXiv

  7. [11]

    Bowman, and Shi Feng

    Arjun Panickssery, Samuel R. Bowman, and Shi Feng. Llm evaluators recognize and favor their own generations, 2024a. URL https://arxiv.org/abs/2404.13076. Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering Llama 2 via Contrastive Activation Addition, July 2024b. URL http://arxiv. org/abs/2312.06681....

    arXiv

  8. [12]

    Avital Shafran, Roei Schuster, Thomas Ristenpart, and Vitaly Shmatikov

    URL https://arxiv.org/abs/2305.18290. Avital Shafran, Roei Schuster, Thomas Ristenpart, and Vitaly Shmatikov. Rerouting LLM Routers, January

    arXiv

Show all 17 references
  1. [13]

    arXiv:2501.01818 [cs]

    URL http://arxiv.org/abs/2501.01818. arXiv:2501.01818 [cs]. Koki Wataoka, Tsubasa Takahashi, and Ryokan Ri. Self-Preference Bias in LLM-as-a-Judge, June

    arXiv

  2. [14]

    arXiv:2410.21819 [cs]

    URL http://arxiv.org/abs/2410.21819. arXiv:2410.21819 [cs]. Martin Weyssow, Aton Kamanda, Xin Zhou, and Houari Sahraoui. Codeultrafeedback: An llm- as-a-judge dataset for aligning large language models to coding preferences. arXiv preprint arXiv:2403.09032,

    arXiv

  3. [15]

    URL http://arxiv.org/abs/2410. 02736. arXiv:2410.02736 [cs]. Tuo Zhang, Asal Mehradfar, Dimitrios Dimitriadis, and Salman Avestimehr. Leveraging Uncertainty Estimation for Efficient LLM Routing, February

    arXiv

  4. [16]

    URL http://arxiv.org/abs/2502. 11021. arXiv:2502.11021 [cs]. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-Judge with MT-Bench...

    arXiv

  5. [17]

    1" or "2

    URL http: //arxiv.org/abs/2306.05685. arXiv:2306.05685 [cs]. 7 A Steering Vector Plots A.1 Illegitimate Self-Preference -0.5 -0.3 -0.1 0.1 0.3 0.5 0 0.25 0.5 0.75 1.0 CAA aware bias Optimization aware bias CAA unaware bias Optimization unaware bias Steering Vector Effect on Out...

    arXiv

  6. [2018]

    URL https:// arxiv.org/abs/1808.08745. OpenAI. ChatGPT (May 24 version) [Large language model],

    arXiv

  7. [2023]

    Yuanpu Cao, Tianrong Zhang, Bochuan Cao, Ziyi Yin, Lu Lin, Fenglong Ma, and Jinghui Chen

    URL https://arxiv.org/abs/2308.06595. Yuanpu Cao, Tianrong Zhang, Bochuan Cao, Ziyi Yin, Lu Lin, Fenglong Ma, and Jinghui Chen. Personalized steering of large language models: Versatile steering vectors through bi-directional preference optimization. Advances in Neural Informa...

    arXiv

  8. [2024]

    Do LLM Evaluators Pre- fer Themselves for a Reason?, April 2025a

    Wei-Lin Chen, Zhepei Wei, Xinyu Zhu, Shi Feng, and Yu Meng. Do LLM Evaluators Pre- fer Themselves for a Reason?, April 2025a. URL http://arxiv.org/abs/2504.03846. arXiv:2504.03846 [cs]. Zhi-Yuan Chen, Hao Wang, Xinyu Zhang, Enrui Hu, and Yankai Lin. Beyond the surface: Measuri...

  9. [2025]

    arXiv:2410.02064 [cs]

    URL http://arxiv.org/abs/2410.02064. arXiv:2410.02064 [cs]. Anthropic. Introducing Claude 3.5 Sonnet,

    arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.