Pith. sign in

REVIEW 3 major objections 4 minor 23 references

Adaptive Repetition for Mitigating Position Bias in LLM-Based Ranking

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A dynamic early-stopping rule for repeated pairwise LLM judgments preserves full-consensus accuracy while cutting average LLM calls by 81 percent, with a confidence-based variant reaching 87 percent savings at a slight accuracy cost.

desk verdict Per-instance early stopping for repeated pairwise LLM judgments is a genuinely useful, simple idea, but the exact convergence claim is oversold and the confidence variant needs more detail. read the letter →

arxiv 2507.17788 v1 pith:PH5M6UVA submitted 2025-07-23 cs.LG cs.AI

Ali Vardasbi , Gustavo Penha , Claudia Hauff , Hugues Bouchard This is my paper
classification cs.LGcs.AI
keywords positionbiasLLM-as-a-judgeearlystoppingmajorityvotingrepetitionconsistencypermutationpairwiserankingconfidenceelicitation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the standard cure for position bias in LLM-as-a-judge tasks, repeating every pairwise judgment many times and taking a majority vote, is wasteful because most pairs do not need all repetitions. It claims that position bias is not only task-dependent but also flips direction from one instance to the next within the same dataset, so a per-instance repetition budget is the right treatment. The paper introduces a dynamic early-stopping rule that stops repeating a pair as soon as majority voting over the two orderings is conclusive, and a confidence-based variant that stops even earlier when the model's stated confidence predicts a large probability gap. On three LLMs and five datasets, it reports matching the accuracy of the full 24-call consensus judgment while cutting the average number of calls by 81 percent, with the confidence variant cutting calls by 87 percent at a slight accuracy cost. If this holds, practitioners can get stable pairwise judgments at roughly a tenth of the inference cost.

What carries the argument

The load-bearing mechanism is Observation 1: with high empirical probability, for each candidate pair and each LLM, at least one of the two orderings $(a,b)$ or $(b,a)$ is repetition-consistent (RC), meaning all $n$ repeated calls with that ordering return the same winner. The early-stopping algorithm uses this to argue that once a majority vote over the two orderings is conclusive, it will stay conclusive for larger $n$, because the RC ordering keeps producing the same verdict. The confidence-based variant adds a second mechanism: the LLM is asked for a confidence value with each judgment, the gap between the average confidence assigned to one candidate versus the other is used to estimate the probability gap $g=|P_a-P_b|$, and the repetition cap for that instance is set to $(1-g)n_M+1$ paired repetitions instead of the full $n_M=12$ paired repetitions.

What would settle it

Run early stopping and the full 24-call consensus on every pair of a dataset whose violation rate for Observation 1 is known and non-trivial, for example a task with many near-tied candidates or a higher sampling temperature, and compare verdicts on the violating pairs specifically; if early stopping disagrees with consensus on a substantial share of those pairs, the claim that it preserves accuracy fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that for almost every candidate pair, at least one of the two input orderings is repetition-consistent, meaning the LLM gives the same verdict every time it is called with that ordering, and that this single fact is enough to stop repetition early. Formally, it defines the consensus outcome $C^{2n}(a,b)$ by majority vote over the concatenated repeated judgments of $(a,b)$ and $(b,a)$, and proposes an early-stopping rule that increments the repetition count $n$ only while the majority is inconclusive, i.e., while $C^{2n}(a,b)$ is a tie. The argument classifies outcomes into four cases, shows that only the case where both orderings are consistent but favor different candidates requires more repetitions, and relies on Observation 1 to conclude that once a majority is conclusive it remains conclusive. Over its experiments, the plain early-stopping method reaches normalized accuracy 1 relative to the full 24-call consensus outcome, while reducing the average number of LLM calls per judgment from 24 to between about 3.1 and 9.4 depending on model and task. The confidence-based refinement reduces calls further, to between about 2.4 and 4.7 on average, with only a slight accuracy trade-off.

Load-bearing premise

The load-bearing premise is that for nearly every candidate pair, at least one of the two orderings yields the same verdict on every repetition, so the small share of pairs where both orderings are unstable (0.7 to 6.3 percent in the paper's experiments) can be treated as negligible.

Editorial extensions

If this is right

  • Evaluation pipelines that currently use a fixed 24-call majority vote for pairwise LLM judgments can cut inference cost by roughly 81 percent without changing judged outcomes, according to the paper's experiments.
  • The confidence-based variant cuts calls by about 87 percent relative to full consensus, making it attractive when compute is constrained and a small disagreement with the full consensus is acceptable.
  • Because bias direction varies within a single dataset, dataset-level corrections such as always discounting the first or last item are suboptimal; per-instance repetition is the more reliable treatment.
  • The same early-stopping logic can be extended to listwise ranking by replacing the two orderings with permutations of the candidate list, with cyclic permutations offered as a linear-cost option.
  • For hard instances where the LLM is unstable, the method automatically spends more repetitions, concentrating compute where it is needed rather than spending it uniformly.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the 0.7 to 6.3 percent violation rates in Table 1 are not obviously negligible in high-stakes evaluation, so the plain early-stopping guarantee is approximate; deployments should measure their own model's violation rate before assuming zero disagreement with full consensus.
  • Editorial inference: the same criterion, stop repeating once one ordering's verdict is stable and the majority is conclusive, could transfer to other LLM sampling tasks such as self-consistency for code generation or factual QA, since it only requires one ordering or one prompt variant to be repeatable.
  • Editorial inference: the confidence-based savings depend on the linear calibration fit on 10 percent of each dataset, so a natural testable extension is to apply the method without any calibration sample, using the raw confidence gap directly as the stopping bound.
  • Editorial inference: the paper's per-instance view of position bias suggests that future work could predict which instances need more repetitions from prompt features alone, potentially eliminating the need for any repeated calls on easy pairs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies position bias and repetition inconsistency in LLM-based pairwise judgments, reports that both the direction and magnitude of position bias vary substantially across instances within the same dataset, and proposes a per-instance early-stopping method for repeated paired judgments. The method stops repeating once the majority vote over both input orderings becomes conclusive, and a confidence-based variant further reduces the number of LLM calls. Experiments across three LLMs and five datasets report an average 81% reduction in LLM calls with the early-stopping method while claiming a normalized accuracy of 1 against the 24-call consensus, and an 87% reduction with the confidence-based variant at a slight accuracy cost.

Significance. If the claims hold, the paper offers a practical and cost-effective way to replace static repetition with adaptive repetition for LLM-as-a-judge tasks, which is relevant to a broad set of applications such as re-ranking and alignment evaluation. The observation that position-bias direction varies per instance (Section 4.1, Table 2, Figure 2) is a useful empirical contribution that strengthens the case for instance-adaptive mitigation. The paper also evaluates across diverse models and datasets and provides a clear comparison against a consensus-judgment skyline. However, the central exactness claim (normalized accuracy 1) currently rests on an unproven prefix-stability assumption, and the confidence-based variant has a questionable worst-case bound and an underspecified training/evaluation split. These issues are load-bearing for the headline conclusions, so the paper requires revision rather than acceptance in its current form.

major comments (3)
  1. [§2.1, Observation 1] The proof that early stopping converges to the 24-call consensus outcome assumes, without evidence, that an ordering which is repetition-consistent (RC) at the stopping time n' remains RC at the final horizon n=12 with the same stable decision, and that the other ordering cannot later accumulate enough opposite verdicts to flip or tie the majority. Observation 1 only guarantees that at n=12 at least one ordering is RC; it is silent about n' < 12. For example, with J^3(a,b)=(a,a,a) and J^3(b,a)=(b,a,b) the method stops at n=3 and outputs a, yet J^12(a,b)=(a,b,b,...) and J^12(b,a)=(b,b,...) gives C^24=b; this configuration is consistent with Observation 1 and is not counted as a violation in Table 1. Consequently the claim that normalized accuracy is exactly 1 (Figure 3) is not supported by the given argument. The authors should either prove and validate the needed prefix-stability assumption or relax the claim and report the actual rate of agreement with the full consensus.
  2. [Table 1, §2.1 case (IV)] Ignoring case (IV) is justified only if its contribution to the accuracy metric is negligible, but Table 1 lists violation rates up to 6.3% (LLM-1 on Emerton-DPO), and the paper does not report how often such instances produce a verdict different from the full consensus. With a violation rate of several percent, the empirical finding of normalized accuracy exactly 1 in all fifteen model–dataset pairs (Figure 3) is not credible without a per-instance breakdown; it may be an artifact of rounding or of excluding ties. Please report the exact agreement rates, including tie outcomes, and discuss the impact of case (IV) separately.
  3. [§2.2, Confidence-Based Early Stopping] There are two problems with the confidence-based variant. First, the bound 'at most (1−g)nM + 1 paired repetitions are needed to guarantee a conclusive result' does not follow from the worst-case argument: if the first (1−g)nM paired judgments are all minority outcomes, a strict majority in the concatenated vector is reached only after more than 2(1−g)nM paired repetitions, not (1−g)nM + 1. Second, the linear model for the confidence gap is fit on a 10% sample, but the text does not state whether those instances are excluded from the evaluation in Figure 3 and Table 3; including them would favorably bias the reported call counts. Please clarify the train/test split and correct or re-derive the bound.
minor comments (4)
  1. [§2, Observation 1] The sentence 'As is seen in this table, the this percentage remains small' contains a typo ('the this').
  2. [Figure 3] Please provide numeric values or error bars for the normalized accuracy bars, and state explicitly how ties in the consensus judgment are handled in the normalization.
  3. [§3, Models] The phrase 'We refer these by LLM-1...' should be 'We refer to these as LLM-1...'.
  4. [§2.2] The phrase 'for sufficiently large nM' is confusing because nM is fixed at 12; clarify the asymptotic intent or remove the phrase.

Circularity Check

0 steps flagged · score 1.0 of 10

No material circularity: the consensus skyline is an internal but independently generated benchmark, and neither early-stopping method renames a fitted parameter as a prediction; the Section 2.1 guarantee relies on an unverified prefix-stability premise, which is a validity gap rather than a definitional circle.

full rationale

Derivation-chain walk: the load-bearing moves are (i) Observation 1 (Section 2, Table 1), (ii) the Section 2.1 case analysis linking early stopping to the 24-call consensus, and (iii) the Section 2.2 confidence-gap linear fit. None is circular by construction. (i) Observation 1 is an original empirical measurement with reported violation rates (0.7-6.3%), not an imported theorem and not a consequence of any definition; the reference list contains no self-citations, so no self-citation chain is load-bearing. (ii) The 'normalized accuracy 1' claim is falsifiable and not definitionally forced: early stopping returns the majority of the strict prefix at the first non-tie step, C^{2n'}, while the skyline is the majority of the full 24-call concatenated vector, C^{24} (Section 2); these are distinct aggregates and can disagree if later calls flip a prefix majority, so the measured agreement in Figure 3 is genuine empirical evidence rather than a tautology. The Section 2.1 proof, however, argues that 'this consensus will remain unchanged (with high probability) as n increases because, according to Observation 1, the ordering will continue to be RC and consistently generate a' -- a dynamic prefix-stability premise that Observation 1, measured statically at n = 12 (Table 1), does not establish; the paper's own caveat ('for the small fraction of violating instances reported in Table 1, our early stopping criterion may not yield the consensus outcome') covers only case (IV), not a later flip inside case (III). That gap is a correctness/proof-strength risk, not a definitional circle: the equality C^{2n'} = C^{24} is neither assumed in the definitions of RC, PC, and C^{2n} nor derivable from them alone. (iii) The confidence-based method fits a linear gap predictor on a 10% split and uses it only to cap repetitions at (1 - g_hat)nM + 1 (Section 2.2); reported training-instance calls are counted at the full 24 ('When reporting the average number of calls (Table 3), we consider the number of calls for this training data to be 2 x nM (24 in our tests)'), so the 87% call reduction is not an in-sample artifact, and accuracy is produced by real LLM judgments rather than by the fitted values. Verdict: no significant circularity; the flagged prefix-stability concern belongs in correctness risk, not circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on an empirical regularity about LLM repetition behavior, a self-consistency benchmark, and a fitted confidence-gap estimator. There are no new physical or conceptual entities: the probability gap and confidence gap are derived quantities estimated from model outputs, not independent postulates.

free parameters (4)
  • Maximum paired repetitions nM = 12 (24 LLM calls)
    Chosen by hand as a practical threshold; the authors state accuracy plateaus beyond 24 calls but do not show this for every model and dataset (Section 3, Parameters).
  • Sampling temperature = 0.1
    Chosen to reduce response variance while allowing diversity; this affects RC and PC estimates and therefore the early-stopping behavior (Section 3, Parameters).
  • Confidence-gap linear model = Slope and intercept not reported
    Fit on 10 percent of each dataset to predict probability gap from confidence gap; the fitted parameters are not disclosed, and Table 3 counts training instances as fixed 24 calls (Section 2.2).
  • Minimum relevance score gap for TREC-DL pairs = 2
    Sampling filter applied to the re-ranking dataset; likely makes pairwise judgments easier and may inflate the reported accuracy and cost-reduction numbers (Section 3, Datasets).
assumptions (4)
  • domain assumption Observation 1 holds: for each candidate pair, the LLM is repetition-consistent for at least one input ordering with probability close to 1.
    This is the load-bearing empirical regularity behind the early-stopping guarantee, introduced in Section 2 and quantified in Table 1.
  • domain assumption The consensus outcome after 24 repetitions is the correct skyline for accuracy.
    All accuracy claims are normalized against this self-consensus rather than against external ground-truth labels (Section 4.2).
  • domain assumption The confidence gap between the two candidates correlates with the probability gap enough for a linear fit.
    The confidence-based stopping method relies on this correlation, asserted in Section 2.2 without calibration analysis.
  • ad hoc to paper Ignoring case (IV), where neither ordering is repetition-consistent, contributes negligibly to accuracy.
    Section 2.1 explicitly sets aside case (IV) using the low empirical violation rates from Table 1, but the impact on the final accuracy claim is not analyzed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adaptive Repetition for Mitigating Position Bias in LLM-Based Ranking." pith.science (2026). https://pith.science/paper/PH5M6UVA

@misc{pith2026250717788,
  author       = {Pith},
  title        = {Pith review of: Adaptive Repetition for Mitigating Position Bias in LLM-Based Ranking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PH5M6UVA}},
  note         = {Machine review of arXiv:2507.17788}
}
read the original abstract

When using LLMs to rank items based on given criteria, or evaluate answers, the order of candidate items can influence the model's final decision. This sensitivity to item positioning in a LLM's prompt is known as position bias. Prior research shows that this bias exists even in large models, though its severity varies across models and tasks. In addition to position bias, LLMs also exhibit varying degrees of low repetition consistency, where repeating the LLM call with the same candidate ordering can lead to different rankings. To address both inconsistencies, a common approach is to prompt the model multiple times with different candidate orderings and aggregate the results via majority voting. However, this repetition strategy, significantly increases computational costs. Extending prior findings, we observe that both the direction -- favoring either the earlier or later candidate in the prompt -- and magnitude of position bias across instances vary substantially, even within a single dataset. This observation highlights the need for a per-instance mitigation strategy. To this end, we introduce a dynamic early-stopping method that adaptively determines the number of repetitions required for each instance. Evaluating our approach across three LLMs of varying sizes and on two tasks, namely re-ranking and alignment, we demonstrate that transitioning to a dynamic repetition strategy reduces the number of LLM calls by an average of 81%, while preserving the accuracy. Furthermore, we propose a confidence-based adaptation to our early-stopping method, reducing LLM calls by an average of 87% compared to static repetition, with only a slight accuracy trade-off relative to our original early-stopping method.

Figures

Figures reproduced from arXiv: 2507.17788 by the authors.

Figure 1
Figure 1. Example outcomes and corresponding RC, PC and [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Histogram of the probability gap g = |Pa − Pb| on the MSMarco dataset for different LLMs. A probability gap of 0 with “Primacy Biased” means that the first candidate in the input is always the outcome. Higher values for probability gap means lower position bias [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. The accuracy of different methods for judgments, normalized by the con [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

23 extracted references · 8 canonical work pages

  1. [1]

    Chen, D., Chen, R., Zhang, S., Liu, Y., Wang, Y., Zhou, H., Zhang, Q., Wan, Y., Zhou, P., Sun, L.: Mllm-as-a-judge: Assessing multimodal llm-as-a-judge with vision-language benchmark (2024),https://arxiv.org/abs/2402.04788

    arXiv 2024

  2. [2]

    https://huggingface.co/datasets/jondurbin/ truthy-dpo-v0.1/tree/main (2023), accessed: 17-Feb-2025

    Durbin, J.: truthy-dpo-v0.1. https://huggingface.co/datasets/jondurbin/ truthy-dpo-v0.1/tree/main (2023), accessed: 17-Feb-2025

    work page 2023

  3. [3]

    https://huggingface.co/datasets/jondurbin/ py-dpo-v0.1/tree/main (2024), accessed: 17-Feb-2025

    Durbin, J.: py-dpo-v0.1. https://huggingface.co/datasets/jondurbin/ py-dpo-v0.1/tree/main (2024), accessed: 17-Feb-2025

    work page 2024

  4. [4]

    Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., Liu, H., Wang, S., Zhang, K., Wang, Y., Gao, W., Ni, L., Guo, J.: A survey on llm-as-a-judge (2024), http://arxiv.org/abs/2411.15594v4

    arXiv 2024

  5. [5]

    org/abs/2411.10915v1

    Guo, Y., Guo, M., Su, J., Yang, Z., Zhu, M., Li, H., Qiu, M., Liu, S.S.: Bias in large language models: Origin, evaluation, and mitigation (2024),http://arxiv. org/abs/2411.10915v1

    arXiv 2024

  6. [6]

    Li, D., Jiang, B., Huang, L., Beigi, A., Zhao, C., Tan, Z., Bhattacharjee, A., Jiang, Y., Chen, C., Wu, T., Shu, K., Cheng, L., Liu, H.: From generation to judgment: Opportunities and challenges of llm-as-a-judge (2024), http://arxiv.org/abs/ 2411.16594v6

    arXiv 2024

  7. [7]

    Li, H., Chen, J., Ai, Q., Chu, Z., Zhou, Y., Dong, Q., Liu, Y.: Calibraeval: Cali- brating prediction distribution to mitigate selection bias in llms-as-judges (2024), http://arxiv.org/abs/2410.15393v1

    arXiv 2024

  8. [8]

    Li, J., Sun, S., Yuan, W., Fan, R.Z., Zhao, H., Liu, P.: Generative judge for eval- uating alignment (2023),https://arxiv.org/abs/2310.05470

    arXiv 2023

Show all 23 references
  1. [9]

    In: Proceedings of the 2024 Con- ference on Empirical Methods in Natural Language Processing

    Li, Z., Wang, C., Ma, P., Wu, D., Wang, S., Gao, C., Liu, Y.: Split and merge: Aligning position biases in llm-based evaluators. In: Proceedings of the 2024 Con- ference on Empirical Methods in Natural Language Processing. pp. 11084–11108 (2024)

    2024

  2. [10]

    https://huggingface.co/datasets/yleo/ emerton_dpo_pairs_judge/tree/main (2024), accessed: 17-Feb-2025

    Léo, Y.: emerton_dpo_pairs_judge. https://huggingface.co/datasets/yleo/ emerton_dpo_pairs_judge/tree/main (2024), accessed: 17-Feb-2025

    2024

  3. [11]

    Mukherjee, S., Mitra, A., Jawahar, G., Agarwal, S., Palangi, H., Awadallah, A.: Orca: Progressive learning from complex explanation traces of gpt-4 (2023),https: //arxiv.org/abs/2306.02707

    2023 arXiv

  4. [12]

    Qin, Z., Jagerman, R., Hui, K., Zhuang, H., Wu, J., Yan, L., Shen, J., Liu, T., Liu, J., Metzler, D., Wang, X., Bendersky, M.: Large language models are effective text rankers with pairwise ranking prompting (2024), http://arxiv.org/abs/2306. 17563v2

    2024

  5. [13]

    Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C.D., Finn, C.: Direct preference optimization: Your language model is secretly a reward model (2024), https://arxiv.org/abs/2305.18290

    2024 arXiv

  6. [14]

    Robinson, J., Rytting, C.M., Wingate, D.: Leveraging large language models for multiple choice question answering (2024),http://arxiv.org/abs/2210.12353v3

    2024 arXiv

  7. [15]

    Shi, L., Ma, C., Liang, W., Ma, W., Vosoughi, S.: Judging the judges: A system- atic study of position bias in llm-as-a-judge (2024),http://arxiv.org/abs/2406. 07791v7

    2024

  8. [16]

    In: Bouamor, H., Pino, J., Bali, K

    Sottana, A., Liang, B., Zou, K., Yuan, Z.: Evaluation metrics in the era of GPT- 4: Reliably evaluating large language models on sequence to sequence tasks. In: Bouamor, H., Pino, J., Bali, K. (eds.) Proceedings of the 2023 Conference on EmpiricalMethodsinNaturalLanguageProces...

    2023

  9. [17]

    Wang, P., Li, L., Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Liu, Q., Liu, T., Sui, Z.: Large language models are not fair evaluators (2024),http://arxiv.org/abs/ 2305.17926v2

    2024 arXiv

  10. [18]

    Wang, Z., Zhang, H., Li, X., Huang, K.H., Han, C., Ji, S., Kakade, S.M., Peng, H., Ji, H.: Eliminating position bias of language models: A mechanistic approach (2024), http://arxiv.org/abs/2407.01100v2

    2024 arXiv

  11. [19]

    Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., Hooi, B.: Can llms express their uncertainty?anempiricalevaluationofconfidenceelicitationinllms(2024), https: //arxiv.org/abs/2306.13063

    2024 arXiv

  12. [20]

    Ye, J., Wang, Y., Huang, Y., Chen, D., Zhang, Q., Moniz, N., Gao, T., Geyer, W., Huang, C., Chen, P.Y., Chawla, N.V., Zhang, X.: Justice or prejudice? quantifying biases in llm-as-a-judge (2024),http://arxiv.org/abs/2410.02736v2

    2024 arXiv

  13. [21]

    Zheng, C., Zhou, H., Meng, F., Zhou, J., Huang, M.: Large language models are not robust multiple choice selectors (2024),http://arxiv.org/abs/2309.03882v4

    2024 arXiv

  14. [22]

    Zhu, B., Frick, E., Wu, T., Zhu, H., Ganesan, K., Chiang, W.L., Zhang, J., Jiao, J.: Starling-7b:Improvinghelpfulnessandharmlessnesswithrlaif.In:FirstConference on Language Modeling (2024)

    2024

  15. [23]

    Zhu, L., Wang, X., Wang, X.: Judgelm: Fine-tuned large language models are scalable judges (2023),https://arxiv.org/abs/2310.17631

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.