Pith. sign in

REVIEW 6 major objections 6 minor 19 references

Positional Bias in Binary Question Answering: How Uncertainty Shapes Model Preferences

T0 review · 6 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Answer uncertainty, not a fixed model trait, controls when large language models fall back on position: bias is nearly absent on clear questions and grows exponentially as options become hard to distinguish.

desk verdict The high-uncertainty accuracy numbers are computed against an arbitrary “correct” label, which breaks the headline exponential claim, but the dataset and the qualitative trend are still worth a serious look. read the letter →

arxiv 2506.23743 v2 pith:YGBBISFU submitted 2025-06-30 cs.CL

Tiziano Labruna , Simone Gallo , Giovanni Da San Martino This is my paper
classification cs.CL
keywords positionalbiasbinaryquestionansweringansweruncertaintylargelanguagemodelsPreferenceFairnessPositionConsistencySQuAD-it-2pairwiseevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper establishes that positional bias in large language models is not a fixed quirk but a response to uncertainty: when choosing between two options is easy, the order in which the options appear hardly matters; when options become hard to distinguish, models default to position. The authors build SQuAD-it-2, an Italian binary QA benchmark with three controlled uncertainty levels, and add two naturally subjective datasets, WebGPT and Winning Arguments. Across five LLMs and two metrics (Preference Fairness and Position Consistency), they find bias nearly absent under low uncertainty and growing exponentially as doubt increases. This reframes positional bias as a property of the task-model pair, so measuring or mitigating it requires controlling for answer uncertainty.

What carries the argument

The load-bearing mechanism is a controlled uncertainty ladder. SQuAD-it-2 takes the same Italian QA instances and creates three versions: Low (context plus a plausible wrong answer), Medium (the same two answers with context removed), and High (two out-of-context distractors, with the original correct answer discarded). For every item and every model, the preferred answer is placed first in one pass and second in another, and the two responses are compared. Two complementary scores carry the quantitative claim: Preference Fairness (|PF|), a normalized measure of how much the model favors one position regardless of content, and Position Consistency (PC), the fraction of items on which the swapped-order response stays the same. WebGPT and Winning Arguments add natural uncertainty through subjective human quality and persuasiveness judgments.

What would settle it

Ask independent annotators to pick the intended 'correct' answer among the two out-of-context options in the High Uncertainty subset of SQuAD-it-2; if they do not agree with the assigned labels above chance, the accuracy values and the exponential growth curve built on those items are measuring arbitrary labels, not model competence.

Watch

Extended reading notes

Core claim

Positional bias in LLMs is conditional rather than fixed: its magnitude tracks how much doubt the model has about which option is correct. On the low-uncertainty SQuAD-it-2 items, the five models keep high accuracy and nearly identical results in both answer orders; when context is removed or replaced by two out-of-context distractors, accuracy gaps between the correct-first and wrong-first conditions widen considerably, and the paper reports the bias growing exponentially with uncertainty. In the high-uncertainty condition one model shifts from 0.648 accuracy with the labelled correct answer first to 0.108 when it is second, illustrating the scale of the effect. The natural datasets show the same pattern, and on Winning Arguments every model systematically performs better when the more persuasive answer appears second, a result the authors interpret as a learned discourse norm. The paper's conclusion is that order-based choice is a second-order fallback strategy that emerges when semantic cues degrade, which makes positional bias a property of the task-model pair rather than of the model alone.

Load-bearing premise

The High Uncertainty condition assumes that one of two randomly generated, out-of-context wrong answers can serve as a ground-truth 'correct' option; if both alternatives are truly interchangeable, the accuracy numbers from that condition have no valid reference point.

Editorial extensions

If this is right

  • Binary-choice accuracy measured in one order underestimates true model performance whenever the two options are hard to distinguish, so uncertainty-aware evaluation should swap option order and report both conditions.
  • The systematic second-option advantage on Winning Arguments implies that in subjective or persuasive domains, a model's pairwise judgments carry a built-in recency preference that averaging over orders will not remove.
  • The sharp rise in invalid responses under high uncertainty means benchmark pipelines need explicit handling of non-binary outputs before PF and PC can be computed reliably.
  • Task difficulty can be ranked by order sensitivity: items with large correct-first versus correct-second accuracy gaps are precisely the items where model judgment is least trustworthy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test the paper leaves open: replacing the three hand-built uncertainty levels with a continuous uncertainty score (model confidence or answer probability gap) should reproduce the same exponential curve if uncertainty is truly the driver.
  • For reward-model training this implies that near-tie preference pairs are the ones most distorted by answer order, so data collection should balance positions within close-scored pairs rather than across the full dataset.
  • Because PC collapses exactly on ambiguous items, a swapped-order consistency check could double as a free proxy for item uncertainty in future benchmark construction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. The paper studies positional bias in binary-choice question answering: whether LLMs systematically favor the first or second presented option independently of content. The authors construct SQuAD-it-2, an Italian benchmark derived from SQuAD-it, with three uncertainty levels: Low (context plus a plausible but incorrect distractor), Medium (the same pairs without context), and High (two out-of-context distractors, with the correct answer discarded). They also evaluate two human-preference datasets, WebGPT and Winning Arguments, which they treat as naturally high-uncertainty conditions. Five models (Llama-3.1-8B, Gemma-3-12B-Q, Gemini-1.5, Gemini-2, Phi4-14B-Q) answer each instance twice with the two options swapped, and positional bias is quantified through the absolute Preference Fairness (|PF|) and Position Consistency (PC). The headline claim is that positional bias is nearly absent under low-uncertainty conditions but grows exponentially when it becomes doubtful which option is correct.

Significance. If the headline result held, the paper would make a useful contribution: it extends pairwise-evaluator position-bias findings (Shi et al.; CALM) to open QA, it targets Italian, an underrepresented language in this literature, it ships a released dataset (SQuAD-it-2), and it evaluates five diverse models including quantized open-weight and proprietary systems. The two-pass flipped-order protocol with PF and PC is conceptually appropriate, the prompt templates and dataset-construction prompts are documented in the appendices, and the 100-instance check of quantized versus full-precision models is good practice. I also note that PC, and the directional PF computed over both passes, are invariant to which out-of-context answer is labeled 'correct,' so the bias metrics themselves are not victims of the arbitrary labeling in the High Uncertainty condition; the ground-truth objection invalidates the accuracy columns and the Section 4 accuracy discussion, not the bias metrics as such.

major comments (6)
  1. [§3.1, App. A.2, Table 1] The High Uncertainty accuracy columns are not well-defined. Section 3.1 defines the High Uncertainty variant as instances (Q, A_ooc^(1), A_ooc^(2)) in which 'both answers are distractors' and 'the task itself loses semantic validity'; Appendix A.2 confirms that the correct answer is discarded and only two OOC distractors remain. Nevertheless, Table 1 reports Correct-first and Wrong-first accuracy for this condition, and Section 4 interprets those numbers (e.g., 'Llama-3.1-8B shows a drastic drop in accuracy when the wrong answer is presented first (from 0.648 to 0.108)'). Computing accuracy requires designating one of two equally wrong answers as 'correct,' and Section 3.2's protocol ('Each instance consists of a prompt X, a preferred answer Apref, and a distractor Adist') provides no rule for that designation. If the opposite OOC answer were labeled 'correct,' all High Uncertainty accuracy values would swap, turning Llama's 0.648/0.108 into 0.108/0.648 and inverting the Section 4 narrative. The High Uncertainty accuracy columns are therefore arbitrary and should be removed, or replaced by ground-truth-free position-choice statistics, with the implications for the paper's claims acknowledged.
  2. [Abstract; §4–5] The claim that positional bias 'grows exponentially' when the correct choice is doubtful appears in the abstract and is echoed qualitatively in Sections 4 and 5, but no quantitative support is provided anywhere. PF and PC are never reported as numbers: Figure 2 is a bar chart without tabulated values or error bars. No numerical scale of 'uncertainty' is defined: the five conditions are only ordinally ordered, and the placement of WebGPT and Winning Arguments at the high end is asserted rather than measured. No exponential model is fit or compared with alternatives (e.g., linear or monotone growth), and no repeated runs control for sampling noise. With five ordinal points, 'exponential' cannot be distinguished from 'monotone and steep' without such analysis. At minimum, the authors must tabulate PF/PC with confidence intervals over repeated runs, define and measure uncertainty per condition (e.g., inter-annotator agreement, model confidence, or answer entropy), and either justify or retract the specific 'exponential' wording.
  3. [§3.2, §4, Table 1] The treatment of invalid responses is unspecified and is potentially the largest confound in the accuracy comparisons. Table 1 reports very large invalid counts: Llama-3.1-8B has 507 then 2026 invalid responses in the Medium Uncertainty condition and 1897 then 1849 in High Uncertainty (out of 7,609 instances), and Phi4-14B-Q has 420 versus 1448 invalid responses in the two High Uncertainty passes. The paper never states whether accuracy, PC, and PF are computed over all instances with invalid responses counted as wrong, over valid responses only, or over instances valid in both passes. Counting invalids as wrong inflates the correct-first/wrong-first gap whenever refusal rates differ by position, as with Phi4's 0.705 versus 0.288 in High Uncertainty, and computing PC on the both-pass-valid subset conditions the metric on model compliance, which varies sharply by condition for the very models (Llama, Gemini) that drive the paper's conclusions. The authors must specify the handling rule for every metric and provide a robustness analysis under alternative treatments.
  4. [§3.2, App. B] The High Uncertainty experiment is not reproducible from the manuscript. Section 3.2's protocol presupposes an Apref/Adist split that, as noted in the first major comment, does not exist for this condition, and no rule is given for assigning the two generated OOC answers to Pass 1 and Pass 2. Appendix B supplies prompt templates for the Low and Medium conditions but omits the High Uncertainty template entirely, even though Table 1 reports High Uncertainty results. The authors should specify the complete High Uncertainty prompt and the OOC-answer-to-position assignment rule, or state explicitly that the assignment is arbitrary and acknowledge the consequences for the accuracy columns.
  5. [App. B, §3.1] The low-to-medium comparison is confounded. The Low and Medium conditions differ not only in the presence of the context but also in the instruction: Low says 'Scegli la risposta corretta' ('choose the correct answer'), while Medium says 'Scegli la risposta che reputi più corretta. Se credi che nessuna sia corretta, scegli comunque quella che reputi più plausibile' (choose the answer you consider most correct, and if you believe none is correct, still choose the most plausible). The latter explicitly licenses guessing and plausibility-based choice, which could by itself increase position-following behavior. The observed Low/Medium differences are therefore not attributable solely to the removal of contextual evidence; the prompt should be held fixed across the two conditions, or a control should be run with identical instructions.
  6. [§3.4.1] The Preference Fairness formula is under-specified for the two-pass protocol. In Shi et al., ipr and irr are dataset-level fractions of instances with the preferred answer in the first and second positions, summing to one; here, every instance is presented twice, once with Apref first and once with Apref second, so the effective ipr and irr values depend on whether instances or presentations are counted, and the paper never says which. Likewise, the normalization constants S−min and S+max are described only abstractly ('under the given conditions') and never evaluated. As a result, PF cannot be recomputed from the manuscript, and since PF is one of the two metrics carrying the headline trend, this is a reproducibility gap in the central measurement.
minor comments (6)
  1. [Figure 1; §3.1] The Figure 1 caption and the Section 3.1 overview state that the SQuAD datasets 'include two answers to a question, one correct and one incorrect,' which contradicts the High Uncertainty construction in which both candidates are incorrect.
  2. [§3.1, §3.3] Gemini-2 generated the plausible distractors and both OOC answers for SQuAD-it-2 and is also one of the five evaluated models; the paper should discuss the potential contamination risk, especially in the High Uncertainty condition where both candidate answers are Gemini-2 outputs, so Gemini-2's 'accuracy' is computed over an arbitrary labeling of its own generations.
  3. [§3.2, §3.3] The decoding configuration is never reported: temperature, sampling strategy, and determinism settings are absent, so with a single run per order the reported PC and PF values carry unknown sampling noise and readers cannot tell whether repetitional consistency would be trivially one.
  4. [§5] The Section 5 explanation of the Winning Arguments second-option preference as internalized discourse norms is speculative; the alternative that the 'Message 1'/'Message 2' prompt format, or the position of the answer field relative to the two messages, contributes to the effect is not tested.
  5. [§4] The Section 4 discussion of the High Uncertainty rows uses accuracy language ('models diverge sharply,' 'strong sensitivity to order') without flagging that the underlying numbers are computed against an arbitrary 'correct' label; this follows from the first major comment and should be revised for consistency.
  6. [§3.1, §4] The paper nowhere defines how 'uncertainty' is operationalized for WebGPT and Winning Arguments: no human-agreement statistics, score-margin distributions, or other evidence shows that these datasets are indeed higher-uncertainty than SQuAD-it-2 Medium, yet their placement at the top of the uncertainty ordering drives the headline trend.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim is an empirical measurement independent of its inputs.

full rationale

The paper's central derivation is empirical: it measures Preference Fairness and Position Consistency across datasets constructed to vary in uncertainty, then reports a qualitative trend. The metrics PF and PC are adopted from Shi et al. (not a self-citation) and are computed from logged model choices, not from any fitted parameter. No parameter is fitted to the target claim, and the paper does not import a uniqueness theorem, ansatz, or load-bearing result from the authors' prior work. The only self-citation is Casola et al. [2] in related work, which is descriptive and not load-bearing for the paper's conclusions. The high-uncertainty SQuAD-it-2 variant discards the correct answer and retains two out-of-context distractors, making the 'accuracy' columns in Table 1 semantically questionable because there is no ground-truth answer by construction. However, this is a construct-validity and reproducibility concern, not a circularity in the derivation: the headline claim about positional bias is based on PF/PC, which do not reduce to the accuracy labels, and no equation in the paper makes the conclusion equivalent to its inputs by definition. The dataset generation and evaluation both involve Gemini-2, which is a contamination risk but not circular reasoning. The derivation chain is therefore self-contained with respect to the paper's own equations.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim rests on a small number of implicit assumptions about what the benchmark variants measure and about the stability of model outputs. No parameters are fitted to data. The most fragile assumption is that accuracy is meaningful in the High Uncertainty variant, where no ground-truth answer exists.

assumptions (3)
  • ad hoc to paper If both candidate answers are randomly generated out-of-context distractors, one of them can still be labeled the 'correct' option for accuracy computation.
    Needed for Table 1 High Uncertainty accuracy; contradicts Section 3.1 statement that both options are distractors.
  • domain assumption The three SQuAD-it-2 variants form a monotonic uncertainty ladder (low, medium, high).
    Section 3.1 assumes removing context and replacing answers with OOC distractors raises uncertainty, but model uncertainty is never measured; prompt wording also changes between Low and Medium.
  • domain assumption Two responses to the same prompt are enough to distinguish position bias from sampling noise.
    Section 3.4 explicitly drops Repetitional Consistency and reports no temperature or seed control, so repeated-query noise is not separated from ordering effects.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Positional Bias in Binary Question Answering: How Uncertainty Shapes Model Preferences." pith.science (2026). https://pith.science/paper/YGBBISFU

@misc{pith2026250623743,
  author       = {Pith},
  title        = {Pith review of: Positional Bias in Binary Question Answering: How Uncertainty Shapes Model Preferences},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YGBBISFU}},
  note         = {Machine review of arXiv:2506.23743}
}
read the original abstract

Positional bias in binary question answering occurs when a model systematically favors one choice over another based solely on the ordering of presented options. In this study, we quantify and analyze positional bias across five large language models under varying degrees of answer uncertainty. We re-adapted the SQuAD-it dataset by adding an extra incorrect answer option and then created multiple versions with progressively less context and more out-of-context answers, yielding datasets that range from low to high uncertainty. Additionally, we evaluate two naturally higher-uncertainty benchmarks: (1) WebGPT - question pairs with unequal human-assigned quality scores, and (2) Winning Arguments - where models predict the more persuasive argument in Reddit's r/ChangeMyView exchanges. Across each dataset, the order of the "correct" (or higher-quality/persuasive) option is systematically flipped (first placed in position 1, then in position 2) to compute both Preference Fairness and Position Consistency. We observe that positional bias is nearly absent under low-uncertainty conditions, but grows exponentially when it becomes doubtful to decide which option is correct.

Figures

Figures reproduced from arXiv: 2506.23743 by the authors.

Figure 1
Figure 1. Overview of the five datasets used in the study, including the settings for each dataset: for the [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Visualization of positional bias across models and datasets, reported by the absolute value of [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 9 canonical work pages

  1. [1]

    Phi- 4-mini technical report: Compact yet power- ful multimodal language models via mixture-of- loras

    Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al. Phi- 4-mini technical report: Compact yet power- ful multimodal language models via mixture-of- loras. arXiv preprint arXiv:2503.01743 , 2025

    arXiv 2025

  2. [2]

    Testing chatgpt for stability and reasoning: A case study using ital- ian medical specialty tests

    Silvia Casola, Tiziano Labruna, Alberto Lavelli, Bernardo Magnini, et al. Testing chatgpt for stability and reasoning: A case study using ital- ian medical specialty tests. In Proceedings of the 9th Italian Conference on Computational Lin- guistics (CLiC-it 2023) , 2023

    work page 2023

  3. [3]

    Neural learning for question an- swering in italian

    Danilo Croce, Alexandra Zelenanska, and Roberto Basili. Neural learning for question an- swering in italian. In Chiara Ghidini, Bernardo Magnini, Andrea Passerini, and Paolo Traverso, editors, AI*IA 2018 – Advances in Artificial In- telligence, pages 389–402, Cham, 2018. Springer International Publishing

    work page 2018

  4. [4]

    Questioning the sur- vey responses of large language models

    Ricardo Dominguez-Olmedo, Moritz Hardt, and Celestine Mendler-D¨ unner. Questioning the sur- vey responses of large language models. Ad- vances in Neural Information Processing Sys- tems, 37:45850–45878, 2024

    work page 2024

  5. [5]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhi- nav Jauhri, Abhinav Pandey, Abhishek Ka- dian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  6. [6]

    Anchored an- swers: Unravelling positional bias in gpt- 2’s multiple-choice questions

    Ruizhe Li and Yanjun Gao. Anchored an- swers: Unravelling positional bias in gpt- 2’s multiple-choice questions. arXiv preprint arXiv:2405.03205, 2024

    arXiv 2024

  7. [7]

    Split and merge: Aligning position biases in llm-based evaluators

    Zongjie Li, Chaozheng Wang, Pingchuan Ma, Daoyuan Wu, Shuai Wang, Cuiyun Gao, and Yang Liu. Split and merge: Aligning position biases in llm-based evaluators. In Yaser Al- Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Process- ing, EMNLP 2024, Miami, FL, USA, November 12-16, ...

    work page 2024

  8. [8]

    Webgpt: Browser- assisted question-answering with human feed- back

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser- assisted question-answering with human feed- back. arXiv preprint arXiv:2112.09332 , 2021

    arXiv 2021

Show all 19 references
  1. [9]

    Squad: 100,000+ 11 questions for machine comprehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ 11 questions for machine comprehension of text. arXiv preprint arXiv:1606.05250 , 2016

    2016 arXiv

  2. [10]

    Judging the judges: A systematic investigation of position bias in pairwise comparative assess- ments by llms

    Lin Shi, Weicheng Ma, and Soroush Vosoughi. Judging the judges: A systematic investigation of position bias in pairwise comparative assess- ments by llms. CoRR, abs/2406.07791, 2024

    2024

  3. [11]

    Winning ar- guments: Interaction dynamics and persuasion strategies in good-faith online discussions

    Chenhao Tan, Vlad Niculae, Cristian Danescu- Niculescu-Mizil, and Lillian Lee. Winning ar- guments: Interaction dynamics and persuasion strategies in good-faith online discussions. In Proceedings of WWW, 2016

    2016

  4. [12]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text. arXiv preprint arXiv:2403.05530 , 2024

    2024 arXiv

  5. [13]

    Gemma 3 technical report

    Gemma Team, Aishwarya Kamath, Johan Fer- ret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram´ e, Morgane Rivi` ere, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025

    2025 arXiv

  6. [14]

    Large language models are not fair eval- uators

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Ling- peng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair eval- uators. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeti...

    2024

  7. [15]

    Eliminating position bias of language models: A mechanistic ap- proach

    Ziqi Wang, Hanlin Zhang, Xiner Li, Kuan-Hao Huang, Chi Han, Shuiwang Ji, Sham M Kakade, Hao Peng, and Heng Ji. Eliminating position bias of language models: A mechanistic ap- proach. arXiv preprint arXiv:2407.01100 , 2024

    2024 arXiv

  8. [16]

    Chawla, and Xiangliang Zhang

    Jiayi Ye, Yanbo Wang, Yue Huang, Dong- ping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V. Chawla, and Xiangliang Zhang. Jus- tice or prejudice? quantifying biases in llm-as-a- judge. In The Thirteenth International Confer- ence on Le...

    2025

  9. [17]

    Large language mod- els are not robust multiple choice selectors

    Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language mod- els are not robust multiple choice selectors. In The Twelfth International Conference on Learn- ing Representations, ICLR 2024, Vienna, Aus- tria, May 7-11, 2024 . OpenReview.net, 2024

    2024

  10. [18]

    Judging llm-as-a-judge with mt- bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt- bench and chatbot arena. Advances in Neu- ral Information Processing Systems , 36:46595– 46623, 2023

    2023

  11. [19]

    Contesto:

    Lianghui Zhu, Xinggang Wang, and Xinlong Wang. Judgelm: Fine-tuned large language models are scalable judges. arXiv preprint arXiv:2310.17631, 2023. A Prompts for SQuAD-it-2 Dataset Generation A.1 Prompt for Low and Medium Un- certainty Settings For the SQuAD-it-2 Low and Medi...

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.