Pith. sign in

REVIEW 3 major objections 5 minor 7 references

Rubric-based LLM-as-a-judge behaves like a multiple-choice test with model-specific position bias: the order of score descriptions shifts the scores a judge assigns, and balanced permutation of the rubric can reveal and partly correct that

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-03 05:25 UTC pith:JSPSVI2B

load-bearing objection Rubric-based LLM judges do show real position bias, and the balanced-permutation idea is worth borrowing; but the mitigation and selection claims rest on a small, non-random permutation set and in-sample validation. the 3 major comments →

arxiv 2602.02219 v2 pith:JSPSVI2B submitted 2026-02-02 cs.CL

Am I More Pointwise or Pairwise? Revealing Position Bias in Rubric-Based LLM-as-a-Judge

classification cs.CL
keywords position biasLLM-as-a-judgerubric-based evaluationbalanced permutationmultiple-choice scoringscore calibrationprompt order effectsevaluation bias
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that when an LLM judges a text against a rubric of score descriptions, the order in which the score options are listed acts as a hidden answer key: the model tends to select options at certain positions regardless of content, so rubric-based evaluation is not a neutral pointwise score but a multiple-choice problem with systematic position bias. Evidence across five models and four datasets shows the bias is consistent within a model but varies between models; all tested judges overuse the first position, and some overuse the last position as well. The paper proposes a balanced permutation scheme: score each item under ten orderings that rotate the rubric options, then average, which cancels out positional preference. This permutation averaging improves correlation with human ratings for most models, and the paper defines a "Bias Cost" for picking a single low-bias ordering when averaging is impractical. If correct, the result recasts rubric-based LLM judging as a measurable and correctable bias rather than a faithful reflection of text quality.

Core claim

Rubric-based LLM-as-a-judge should not be treated as pointwise scoring; because the model selects one of several listed score descriptions, it is implicitly a multiple-choice selection. The paper's controlled experiments show that selection probability by position deviates systematically from the uniform 20% baseline, with all tested models preferring the first position and many preferring the last; the direction and strength of this bias are model-specific and consistent across datasets. The balanced permutation set of ten orderings makes score identity and position orthogonal, so averaging scores across them reveals the latent position bias and, for most models, yields gains in Spearman co

What carries the argument

Balanced permutation: a set of ten rubric orderings (five forward cyclic rotations and five reverse cyclic rotations of the five score descriptions) in which every score appears exactly twice at each position. Marginalizing over this set factors out score identity from position, so any remaining nonuniform selection by position is a readout of position bias; averaging the ten predictions yields a debiased score. The Bias Cost, defined as the sum over positions of the absolute deviation of the selection probability from 20%, serves as a human-free criterion for choosing a single low-bias ordering.

Load-bearing premise

The experiments assume that ten cyclic rotations of the rubric list are enough to isolate position effects from other properties of the ordering (like whether the list ascends or descends); if a full set of 120 permutations changes the measured pattern, the four-dataset conclusion would need revision.

What would settle it

Run the same judge on the same items under all 120 permutations of the five rubric descriptions and compare the position selection distribution; if the first-position preference seen under cyclic rotations does not persist for the full permutation set, the claimed position bias is an artifact of the rotation sampling rather than a stable property of the judge.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A single rubric ordering silently injects a model-specific position bias into any benchmark score, so rubric-based judge scores should not be interpreted as pure content judgments.
  • Averaging over balanced permutations is a cheap, model-agnostic calibration that improves Spearman correlation with human ratings for most tested models, with gains up to +.089 on the story benchmark.
  • The Bias Cost lets a practitioner pick a low-bias rubric ordering using only a short probe of the judge model, with no human annotations.
  • Rubric order can be deliberately manipulated to shift judge outputs, implying that evaluation transparency and any process that audits LLM-assisted assessments should treat rubric order as a potentially exploitable signal.
  • Permuting rubric orderings during training when rubrics are used as training signals may improve training stability, a hypothesis the paper puts forward.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The ten cyclic rotations are a thin sample of the 120 possible permutations, so the 20% null test conflates raw position with other ordering properties such as ascending/descending list direction or adjacency between adjacent score descriptions; a full-permutation control would separate those effects and could change the measured bias magnitude.
  • The primacy and recency pattern echoes long-context processing known from other LLM behaviors, suggesting the same mechanism may extend to any ordered list of choices, such as Likert-scale items or survey response options; this is an inference beyond the paper's data.
  • Because permutation averaging helps models with strong bias but can hurt a model that happened to perform well with one favorable ordering, the paper's results imply that optimal practice is model-dependent: some judges are better calibrated by repeating a fixed ordering, others by shuffling.
  • A concrete testable extension: run the same judge over all 120 permutations on a single dataset and check whether the first-position preference persists; if it disappears, the bias reported here may be an artifact of the cyclic-rotation design rather than a stable property of the judge.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies position bias in rubric-based LLM-as-a-judge, where the model selects a score from a list of five ordered rubric descriptions. The authors propose a balanced-permutation method that uses 10 cyclic rotations (5 forward, 5 reverse) so that each score appears equally often at each position. Aggregating selections across these permutations, they report that model choices deviate from the 20% per-position baseline, concluding that rubric-based evaluation exhibits position bias. They also test whether averaging scores across permutations improves correlation with human judgments (compared with repeating a fixed ordering), and they introduce a Bias Cost heuristic for selecting a rubric ordering without human labels. Experiments cover 7 model configurations and 4 datasets, two with human annotations.

Significance. If the central existence claim holds, this is a useful and timely result: rubric-based evaluation is widely used for LLM alignment and benchmarking, and showing that the order of score descriptions systematically shifts scores would have practical consequences. The balanced-permutation design is a simple, model-agnostic idea, and the paper goes beyond existence by testing a mitigation strategy on human-annotated data. The breadth of models and datasets is a strength, and the focus on a less-studied evaluation paradigm differentiates it from prior pairwise/pointwise bias work. However, the current evidence for 'position bias' per se is weakened by the very restricted permutation set, and the Bias Cost proposal is validated in-sample; these issues must be addressed before the claims can be accepted.

major comments (3)
  1. [§3.1 / Figure 5] The balanced-permutation design uses only the 10 cyclic rotations of [1,2,3,4,5]. This set does not vary linear adjacency or monotone-order structure: the natural consecutive pairs (1,2), (2,3), etc. remain adjacent in all 10 orderings, and every permutation is either a forward or reverse cyclic shift. Therefore the observed deviation from the 20% null could conflate genuine position preference with other list-order effects (e.g., a preference for monotone ascending/descending lists, or a sensitivity to the adjacency of particular score descriptions). The 20% null is correct for position marginals under this set, but the generalization from 'bias within cyclic rotations' to 'position bias in rubric-based evaluation' requires a control with non-cyclic or randomly sampled permutations. Please run such a control (or provide a formal argument that the cyclic set is sufficient) before claimin
  2. [§5.3 / Table 6] The Bias Cost analysis is in-sample and therefore circular as stated. The probabilities P(p|score) in Table 5 are estimated on the full dataset, and the same data are then used to select the min-bias ordering and to compute that ordering's correlation rank in Table 6. This only shows a post-hoc association, not that Bias Cost can predict which ordering will perform best out-of-sample. The proposed procedure in the text ('first probe the model using a small set of rubric orderings... then select') is not actually evaluated. To support the contribution, split the data: estimate the score-position statistics on a probe subset, select an ordering, and evaluate its correlation on held-out items. Without such validation, the claim that Bias Cost identifies reliable orderings is unsupported.
  3. [Abstract vs §5.1] The abstract states 'some judges favor the first option, while others favor the last,' but §5.1 says 'All models exhibit a consistent preference for the first position,' and the Discussion refers to 'primacy and recency effects.' These statements are mutually inconsistent. If some models show a last-position preference, the evidence should be displayed (e.g., in Figure 5 or Table 5); if all models actually prefer the first position, the abstract and the primacy/recency description should be revised. The direction of the bias is a load-bearing part of the claimed model-specificity, so this inconsistency must be resolved.
minor comments (5)
  1. [Figure 2] The illustrative figure uses three score positions ('1st 2nd 3rd') while the experiments use five. This is confusing; please align the schematic with the actual setup.
  2. [Abstract] The abstract contains a sentence fragment: 'While prior works mainly focus on point-wise and pair-wise evaluation paradigms. Rubric-based evaluation...' Please rephrase.
  3. [Table 1] Dataset names are inconsistent: 'MT Bench' vs 'MT-Bench'. Use one form throughout.
  4. [§4.4] The sentence 'we use ten permutations and compare the average score over these ten permutations with the score obtained by directly averaging the ten individual predictions' is confusing because these two averages appear identical. Clarify that the comparison is between averaging across different orderings (Permutation) and averaging repeated runs with a fixed ordering (Repeat).
  5. [Table 5] The table lacks sample sizes and confidence intervals. Since some deviations from 20% are small (e.g., OSS-120B rows are close to 20%), a statistical test or bootstrap CI would strengthen the claim that these deviations are not noise.

Circularity Check

1 steps flagged

Core position-bias finding is not circular, but the Bias Cost ordering-selection result is validated on the same data used to fit the cost, so that secondary claim is partially circular.

specific steps
  1. fitted input called prediction [Sec. 5.3, Bias Cost equation and Table 6]
    "We expect that a lower Bias Cost will lead to higher correlation. As shown in Table 6, we compute the Bias Cost for the ten different rubric orderings in our existing data. In most cases, the ordering with the lowest Bias Cost ranks within the top 50% in terms of correlation, suggesting that this method is effective."

    The Bias Cost is defined from empirical score-position probabilities P(p|score) estimated on the full dataset (Table 5). The 'Corr Rank' in Table 6 is the Pearson correlation of the same full dataset for the same ten orderings. The paper describes a probe/small-set procedure but does not split the data; min-cost orderings are selected and evaluated on the data that produced the cost. Hence the claimed effectiveness is an in-sample selection artifact: the cost and the correlation are functions of the same fitted P(p|score), so the evaluation is not an independent prediction. This is a fitted input presented as a validated prediction.

full rationale

The central claim that rubric-based judges exhibit position bias is not circular. The balanced-permutation design guarantees each score appears exactly twice at each position, so the 20% null in Figure 5 is valid by construction and deviations are empirical evidence. Table 2's permutation-vs-repeat comparison is also experimentally grounded rather than derived from the method's assumptions. Self-citations are introductory and not load-bearing. The only significant circularity is the Bias Cost validation: score-position probabilities are fitted on the full data and the min-cost ordering is then ranked on the same full data, making Table 6 an in-sample evaluation. This weakens the third contribution but does not invalidate the existence claim. The limitation that only 10 cyclic rotations are used is a generalizability/confounding concern, not a by-construction circularity. Score 4 reflects partial circularity in a secondary result.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 0 invented entities

No fitted physical parameters or invented entities. The empirical design has two hand-chosen settings (temperature, number of permutations) that affect all results. The bias-cost threshold 0.2 is the uniform null, not a fitted value. The core claims rest on the domain assumption that 10 cyclic rotations isolate position bias and that human ratings are ground truth.

free parameters (2)
  • temperature for Qwen3 reasoning mode = 0.3
    Set to 0.3 in §4.2 to avoid repetitive loops; affects all Qwen3-Think results, including variance and correlation numbers.
  • number of cyclic permutations = 10
    Five forward and five reverse cyclic rotations; chosen so each score appears exactly twice per position. The paper claims a small number suffices but never sweeps this count.
axioms (3)
  • domain assumption Human annotations on HANNA and SummEval are reliable ground truth for evaluation quality.
    Used throughout §5.2 to compute judge-human correlation; the paper assumes 3-rater aggregates are the correct target.
  • domain assumption Ten cyclic rotations suffice to cancel score-value effects and reveal pure position bias.
    Central to §3.1 and Figure 2; not verified against the full permutation space, so order-direction or adjacency confounds are not excluded.
  • domain assumption The judge model's output is a faithful score selection from the provided rubric.
    The protocol parses 'Feedback: [RESULT] X' as the chosen score; reasoning-mode outputs at temperature 0.3 are treated as parseable and stable.

pith-pipeline@v1.3.0-alltime-deepseek · 14071 in / 14584 out tokens · 134742 ms · 2026-08-03T05:25:13.559700+00:00 · methodology

0 comments
read the original abstract

Large language models are widely employed as evaluators, a paradigm commonly referred to as LLM-as-a-judge. Prior research has predominantly examined point-wise or pair-wise evaluation protocols; in contrast, our focus is on rubric-based evaluation, which has been attracting increasing attention owing to its utility for training models in domains where verification is otherwise difficult. In this work, we show that rubric-based evaluation implicitly resembles a multiple-choice setting and therefore exhibits position bias: LLMs tend to prefer score options that appear at specific positions within the rubric list. Through controlled experiments across multiple models and datasets, we demonstrate that this position bias is consistent. Its direction, however, is model-specific: some judges favor the first option, while others favor the last. We further identify a second, orthogonal axis of bias: when a prompt scores several criteria simultaneously, the ordering of the criteria itself shifts the resulting scores. We additionally explore permuting the order of the rubric options as a means of mitigating position bias, and find that although the bias can be attenuated, improvements in the correlation between model judgments and human annotations are obtained primarily for models that exhibit strong bias. Our results recast rubric-based LLM-as-a-judge as a multiple-choice problem with measurable, model-specific position bias, and we further confirm that only a small number of random order permutations are sufficient to reduce the error introduced by this bias for the majority of models.

Figures

Figures reproduced from arXiv: 2602.02219 by Tadashi Kozuno, Tosho Hirasawa, Yoshitaka Ushiku, Yuzheng Xu.

Figure 1
Figure 1. Figure 1: Three paradigms of LLM-as-a-Judge evaluation. Point￾wise evaluation assigns a score given a question and a single re￾sponse. Pair-wise evaluation compares two responses and outputs the model’s preference. Rubric-based evaluation further incor￾porates explicit scoring criteria. Judge models may also exhibit position bias toward responses in certain orders. a wide range of applications. Although LLMs are hig… view at source ↗
Figure 2
Figure 2. Figure 2: Balanced permutation of rubric orderings. Aggregating the model’s choice distributions across permutations marginalizes out score identities and reveals systematic position bias. also a widely adopted approach (Kim et al., 2024), as it provides greater flexibility. In this paper, we focus on the rubric-based LLM-as-a-Judge setting. Since there is only a single text to be evaluated, there should be no posit… view at source ↗
Figure 4
Figure 4. Figure 4: An illustration of balanced permutations of rubric scores, ensuring that each score appears equally often at each position. Through experiments with balanced permutations, we can confirm the existence of position bias. We also want to ex￾amine whether balanced permutation can reduce the impact of this bias. A straightforward approach is to average the scores across permutations; since the correct score sho… view at source ↗
Figure 3
Figure 3. Figure 3: Unified prompt format used in all experiments (Prometheus-Eval) [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Position selection distribution on 4 datasets. The dashed red line indicates the expected 20% baseline under no position bias. correct score at each position uniformly. All models exhibit a consistent preference for the first position. The severity of the position bias varies largely across mod￾els. Smaller models show stronger bias: Qwen3-8B selects Position 1 in about 30–39% of cases, while larger models… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

7 extracted references · 3 linked inside Pith

  1. [5]

    ijcnlp-long.18/

    URL https://aclanthology.org/2025. ijcnlp-long.18/. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V ., Goyal, N., Hartshorn, A., Hosseini,...

  2. [86]

    emnlp-main.86/

    URL https://aclanthology.org/2025. emnlp-main.86/. Chhun, C., Suchanek, F. M., and Clavel, C. Do language models enjoy their own stories? prompting large language models for automatic story evaluation.Transactions of the Association for Computational Linguistics, 12:1122– 1142, 09 2024. ISSN 2307-387X. doi: 10.1162/tacl a 00689. URL https://doi.org/10.116...

  3. [306]

    findings-acl.306/

    URL https://aclanthology.org/2025. findings-acl.306/. Kim, S., Shin, J., Cho, Y ., Jang, J., Longpre, S., Lee, H., Yun, S., Shin, S., Kim, S., Thorne, J., and Seo, M. Prometheus: Inducing fine-grained evaluation capability in language models. InThe Twelfth International Confer- ence on Learning Representations, 2024. URL https: //openreview.net/forum?id=8...

  4. [365]

    naacl-long.365/

    URL https://aclanthology.org/2024. naacl-long.365/. Goyal, T., Li, J. J., and Durrett, G. News summarization and evaluation in the era of gpt-3.arXiv preprint arXiv: 2209.12356, 2022. Gunjal, A., Wang, A., Lau, E., Nath, V ., He, Y ., Liu, B., and Hendryx, S. Rubrics as rewards: Reinforcement learning beyond verifiable domains.arXiv preprint arXiv: 2507.1...

  5. [2022]

    ISBN 9781713871088

    Curran Associates Inc. ISBN 9781713871088. Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y ., Dirani, J., Michael, J., and Bowman, S. R. GPQA: A graduate-level google-proof q&a benchmark. InFirst Conference on Language Modeling, 2024. URL https: //openreview.net/forum?id=Ti67584b98. Shi, L., Ma, C., Liang, W., Diao, X., Ma, W., and V osoughi...

  6. [2024]

    Li, Q., Dou, S., Shao, K., Chen, C., and Hu, H

    URL https://proceedings.mlr.press/ v235/lee24t.html. Li, Q., Dou, S., Shao, K., Chen, C., and Hu, H. Evaluating scoring bias in llm-as-a-judge.arXiv preprint arXiv: 2506.22316, 2025. Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. Lost in the middle: How language models use long contexts.Transactions of the Assoc...

  7. [2025]

    ISBN 8 Submission and Formatting Instructions 979-8-89176-256-5

    Association for Computational Linguistics. ISBN 8 Submission and Formatting Instructions 979-8-89176-256-5. doi: 10.18653/v1/2025.findings-acl