REVIEW 3 major objections 5 minor 7 references
Rubric-based LLM-as-a-judge behaves like a multiple-choice test with model-specific position bias: the order of score descriptions shifts the scores a judge assigns, and balanced permutation of the rubric can reveal and partly correct that
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 05:25 UTC pith:JSPSVI2B
load-bearing objection Rubric-based LLM judges do show real position bias, and the balanced-permutation idea is worth borrowing; but the mitigation and selection claims rest on a small, non-random permutation set and in-sample validation. the 3 major comments →
Am I More Pointwise or Pairwise? Revealing Position Bias in Rubric-Based LLM-as-a-Judge
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Rubric-based LLM-as-a-judge should not be treated as pointwise scoring; because the model selects one of several listed score descriptions, it is implicitly a multiple-choice selection. The paper's controlled experiments show that selection probability by position deviates systematically from the uniform 20% baseline, with all tested models preferring the first position and many preferring the last; the direction and strength of this bias are model-specific and consistent across datasets. The balanced permutation set of ten orderings makes score identity and position orthogonal, so averaging scores across them reveals the latent position bias and, for most models, yields gains in Spearman co
What carries the argument
Balanced permutation: a set of ten rubric orderings (five forward cyclic rotations and five reverse cyclic rotations of the five score descriptions) in which every score appears exactly twice at each position. Marginalizing over this set factors out score identity from position, so any remaining nonuniform selection by position is a readout of position bias; averaging the ten predictions yields a debiased score. The Bias Cost, defined as the sum over positions of the absolute deviation of the selection probability from 20%, serves as a human-free criterion for choosing a single low-bias ordering.
Load-bearing premise
The experiments assume that ten cyclic rotations of the rubric list are enough to isolate position effects from other properties of the ordering (like whether the list ascends or descends); if a full set of 120 permutations changes the measured pattern, the four-dataset conclusion would need revision.
What would settle it
Run the same judge on the same items under all 120 permutations of the five rubric descriptions and compare the position selection distribution; if the first-position preference seen under cyclic rotations does not persist for the full permutation set, the claimed position bias is an artifact of the rotation sampling rather than a stable property of the judge.
If this is right
- A single rubric ordering silently injects a model-specific position bias into any benchmark score, so rubric-based judge scores should not be interpreted as pure content judgments.
- Averaging over balanced permutations is a cheap, model-agnostic calibration that improves Spearman correlation with human ratings for most tested models, with gains up to +.089 on the story benchmark.
- The Bias Cost lets a practitioner pick a low-bias rubric ordering using only a short probe of the judge model, with no human annotations.
- Rubric order can be deliberately manipulated to shift judge outputs, implying that evaluation transparency and any process that audits LLM-assisted assessments should treat rubric order as a potentially exploitable signal.
- Permuting rubric orderings during training when rubrics are used as training signals may improve training stability, a hypothesis the paper puts forward.
Where Pith is reading between the lines
- The ten cyclic rotations are a thin sample of the 120 possible permutations, so the 20% null test conflates raw position with other ordering properties such as ascending/descending list direction or adjacency between adjacent score descriptions; a full-permutation control would separate those effects and could change the measured bias magnitude.
- The primacy and recency pattern echoes long-context processing known from other LLM behaviors, suggesting the same mechanism may extend to any ordered list of choices, such as Likert-scale items or survey response options; this is an inference beyond the paper's data.
- Because permutation averaging helps models with strong bias but can hurt a model that happened to perform well with one favorable ordering, the paper's results imply that optimal practice is model-dependent: some judges are better calibrated by repeating a fixed ordering, others by shuffling.
- A concrete testable extension: run the same judge over all 120 permutations on a single dataset and check whether the first-position preference persists; if it disappears, the bias reported here may be an artifact of the cyclic-rotation design rather than a stable property of the judge.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies position bias in rubric-based LLM-as-a-judge, where the model selects a score from a list of five ordered rubric descriptions. The authors propose a balanced-permutation method that uses 10 cyclic rotations (5 forward, 5 reverse) so that each score appears equally often at each position. Aggregating selections across these permutations, they report that model choices deviate from the 20% per-position baseline, concluding that rubric-based evaluation exhibits position bias. They also test whether averaging scores across permutations improves correlation with human judgments (compared with repeating a fixed ordering), and they introduce a Bias Cost heuristic for selecting a rubric ordering without human labels. Experiments cover 7 model configurations and 4 datasets, two with human annotations.
Significance. If the central existence claim holds, this is a useful and timely result: rubric-based evaluation is widely used for LLM alignment and benchmarking, and showing that the order of score descriptions systematically shifts scores would have practical consequences. The balanced-permutation design is a simple, model-agnostic idea, and the paper goes beyond existence by testing a mitigation strategy on human-annotated data. The breadth of models and datasets is a strength, and the focus on a less-studied evaluation paradigm differentiates it from prior pairwise/pointwise bias work. However, the current evidence for 'position bias' per se is weakened by the very restricted permutation set, and the Bias Cost proposal is validated in-sample; these issues must be addressed before the claims can be accepted.
major comments (3)
- [§3.1 / Figure 5] The balanced-permutation design uses only the 10 cyclic rotations of [1,2,3,4,5]. This set does not vary linear adjacency or monotone-order structure: the natural consecutive pairs (1,2), (2,3), etc. remain adjacent in all 10 orderings, and every permutation is either a forward or reverse cyclic shift. Therefore the observed deviation from the 20% null could conflate genuine position preference with other list-order effects (e.g., a preference for monotone ascending/descending lists, or a sensitivity to the adjacency of particular score descriptions). The 20% null is correct for position marginals under this set, but the generalization from 'bias within cyclic rotations' to 'position bias in rubric-based evaluation' requires a control with non-cyclic or randomly sampled permutations. Please run such a control (or provide a formal argument that the cyclic set is sufficient) before claimin
- [§5.3 / Table 6] The Bias Cost analysis is in-sample and therefore circular as stated. The probabilities P(p|score) in Table 5 are estimated on the full dataset, and the same data are then used to select the min-bias ordering and to compute that ordering's correlation rank in Table 6. This only shows a post-hoc association, not that Bias Cost can predict which ordering will perform best out-of-sample. The proposed procedure in the text ('first probe the model using a small set of rubric orderings... then select') is not actually evaluated. To support the contribution, split the data: estimate the score-position statistics on a probe subset, select an ordering, and evaluate its correlation on held-out items. Without such validation, the claim that Bias Cost identifies reliable orderings is unsupported.
- [Abstract vs §5.1] The abstract states 'some judges favor the first option, while others favor the last,' but §5.1 says 'All models exhibit a consistent preference for the first position,' and the Discussion refers to 'primacy and recency effects.' These statements are mutually inconsistent. If some models show a last-position preference, the evidence should be displayed (e.g., in Figure 5 or Table 5); if all models actually prefer the first position, the abstract and the primacy/recency description should be revised. The direction of the bias is a load-bearing part of the claimed model-specificity, so this inconsistency must be resolved.
minor comments (5)
- [Figure 2] The illustrative figure uses three score positions ('1st 2nd 3rd') while the experiments use five. This is confusing; please align the schematic with the actual setup.
- [Abstract] The abstract contains a sentence fragment: 'While prior works mainly focus on point-wise and pair-wise evaluation paradigms. Rubric-based evaluation...' Please rephrase.
- [Table 1] Dataset names are inconsistent: 'MT Bench' vs 'MT-Bench'. Use one form throughout.
- [§4.4] The sentence 'we use ten permutations and compare the average score over these ten permutations with the score obtained by directly averaging the ten individual predictions' is confusing because these two averages appear identical. Clarify that the comparison is between averaging across different orderings (Permutation) and averaging repeated runs with a fixed ordering (Repeat).
- [Table 5] The table lacks sample sizes and confidence intervals. Since some deviations from 20% are small (e.g., OSS-120B rows are close to 20%), a statistical test or bootstrap CI would strengthen the claim that these deviations are not noise.
Circularity Check
Core position-bias finding is not circular, but the Bias Cost ordering-selection result is validated on the same data used to fit the cost, so that secondary claim is partially circular.
specific steps
-
fitted input called prediction
[Sec. 5.3, Bias Cost equation and Table 6]
"We expect that a lower Bias Cost will lead to higher correlation. As shown in Table 6, we compute the Bias Cost for the ten different rubric orderings in our existing data. In most cases, the ordering with the lowest Bias Cost ranks within the top 50% in terms of correlation, suggesting that this method is effective."
The Bias Cost is defined from empirical score-position probabilities P(p|score) estimated on the full dataset (Table 5). The 'Corr Rank' in Table 6 is the Pearson correlation of the same full dataset for the same ten orderings. The paper describes a probe/small-set procedure but does not split the data; min-cost orderings are selected and evaluated on the data that produced the cost. Hence the claimed effectiveness is an in-sample selection artifact: the cost and the correlation are functions of the same fitted P(p|score), so the evaluation is not an independent prediction. This is a fitted input presented as a validated prediction.
full rationale
The central claim that rubric-based judges exhibit position bias is not circular. The balanced-permutation design guarantees each score appears exactly twice at each position, so the 20% null in Figure 5 is valid by construction and deviations are empirical evidence. Table 2's permutation-vs-repeat comparison is also experimentally grounded rather than derived from the method's assumptions. Self-citations are introductory and not load-bearing. The only significant circularity is the Bias Cost validation: score-position probabilities are fitted on the full data and the min-cost ordering is then ranked on the same full data, making Table 6 an in-sample evaluation. This weakens the third contribution but does not invalidate the existence claim. The limitation that only 10 cyclic rotations are used is a generalizability/confounding concern, not a by-construction circularity. Score 4 reflects partial circularity in a secondary result.
Axiom & Free-Parameter Ledger
free parameters (2)
- temperature for Qwen3 reasoning mode =
0.3
- number of cyclic permutations =
10
axioms (3)
- domain assumption Human annotations on HANNA and SummEval are reliable ground truth for evaluation quality.
- domain assumption Ten cyclic rotations suffice to cancel score-value effects and reveal pure position bias.
- domain assumption The judge model's output is a faithful score selection from the provided rubric.
read the original abstract
Large language models are widely employed as evaluators, a paradigm commonly referred to as LLM-as-a-judge. Prior research has predominantly examined point-wise or pair-wise evaluation protocols; in contrast, our focus is on rubric-based evaluation, which has been attracting increasing attention owing to its utility for training models in domains where verification is otherwise difficult. In this work, we show that rubric-based evaluation implicitly resembles a multiple-choice setting and therefore exhibits position bias: LLMs tend to prefer score options that appear at specific positions within the rubric list. Through controlled experiments across multiple models and datasets, we demonstrate that this position bias is consistent. Its direction, however, is model-specific: some judges favor the first option, while others favor the last. We further identify a second, orthogonal axis of bias: when a prompt scores several criteria simultaneously, the ordering of the criteria itself shifts the resulting scores. We additionally explore permuting the order of the rubric options as a means of mitigating position bias, and find that although the bias can be attenuated, improvements in the correlation between model judgments and human annotations are obtained primarily for models that exhibit strong bias. Our results recast rubric-based LLM-as-a-judge as a multiple-choice problem with measurable, model-specific position bias, and we further confirm that only a small number of random order permutations are sufficient to reduce the error introduced by this bias for the majority of models.
Figures
Reference graph
Works this paper leans on
-
[5]
URL https://aclanthology.org/2025. ijcnlp-long.18/. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V ., Goyal, N., Hartshorn, A., Hosseini,...
Pith/arXiv arXiv 2025
-
[86]
URL https://aclanthology.org/2025. emnlp-main.86/. Chhun, C., Suchanek, F. M., and Clavel, C. Do language models enjoy their own stories? prompting large language models for automatic story evaluation.Transactions of the Association for Computational Linguistics, 12:1122– 1142, 09 2024. ISSN 2307-387X. doi: 10.1162/tacl a 00689. URL https://doi.org/10.116...
doi:10.1162/tacl 2025
-
[306]
findings-acl.306/
URL https://aclanthology.org/2025. findings-acl.306/. Kim, S., Shin, J., Cho, Y ., Jang, J., Longpre, S., Lee, H., Yun, S., Shin, S., Kim, S., Thorne, J., and Seo, M. Prometheus: Inducing fine-grained evaluation capability in language models. InThe Twelfth International Confer- ence on Learning Representations, 2024. URL https: //openreview.net/forum?id=8...
2025
-
[365]
URL https://aclanthology.org/2024. naacl-long.365/. Goyal, T., Li, J. J., and Durrett, G. News summarization and evaluation in the era of gpt-3.arXiv preprint arXiv: 2209.12356, 2022. Gunjal, A., Wang, A., Lau, E., Nath, V ., He, Y ., Liu, B., and Hendryx, S. Rubrics as rewards: Reinforcement learning beyond verifiable domains.arXiv preprint arXiv: 2507.1...
Pith/arXiv arXiv 2024
-
[2022]
ISBN 9781713871088
Curran Associates Inc. ISBN 9781713871088. Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y ., Dirani, J., Michael, J., and Bowman, S. R. GPQA: A graduate-level google-proof q&a benchmark. InFirst Conference on Language Modeling, 2024. URL https: //openreview.net/forum?id=Ti67584b98. Shi, L., Ma, C., Liang, W., Diao, X., Ma, W., and V osoughi...
2024
-
[2024]
Li, Q., Dou, S., Shao, K., Chen, C., and Hu, H
URL https://proceedings.mlr.press/ v235/lee24t.html. Li, Q., Dou, S., Shao, K., Chen, C., and Hu, H. Evaluating scoring bias in llm-as-a-judge.arXiv preprint arXiv: 2506.22316, 2025. Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. Lost in the middle: How language models use long contexts.Transactions of the Assoc...
Pith/arXiv arXiv 2025
-
[2025]
ISBN 8 Submission and Formatting Instructions 979-8-89176-256-5
Association for Computational Linguistics. ISBN 8 Submission and Formatting Instructions 979-8-89176-256-5. doi: 10.18653/v1/2025.findings-acl
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.