Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

SCOPE: Stochastic and Counterbiased Option Placement for Evaluating Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By measuring each model's position bias with meaningless prompts and placing correct answers inversely to it, SCOPE caps the lucky-rate at $1/n$, so any score above that threshold must reflect genuine understanding.

desk verdict The inverse-bias placement bound is correct as an inequality, but the paper misstates it, overclaims the corollary, and never validates the null-prompt-to-real-item transfer that the method depends on. read the letter →

arxiv 2507.18182 v2 pith:PNVWP2QJ submitted 2025-07-24 cs.CL cs.AI

Wonjun Jeong , Dongseok Kim , Taegkeun Whangbo This is my paper
classification cs.CLcs.AI
keywords selectionbiaspositionmultiple-choiceevaluationlucky-rateinverse-biasplacementnullpromptsemanticdistractorLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that much of the apparent accuracy of language models on multiple-choice tests is a by-product of where the correct answer happens to sit, and that this artifact can be removed structurally. It proposes SCOPE, a two-module evaluation protocol: a null prompt with no semantic content is repeated to measure the model's position bias, and the correct answer is then placed in slots sampled inversely to that bias. The central theoretical claim is that this inverse placement caps the lucky-rate—the probability of getting a question right from position alone—at $1/n$. The paper also spreads the most semantically similar wrong option far from the correct one, and argues that with these two guards in place, correct answers reflect genuine understanding rather than shortcut strategies.

What carries the argument

The load-bearing object is the inverse-bias distribution $Q$, defined by $q_i = (1/p_i)/\sum_{j=1}^n 1/p_j$, built from the null-prompt position-bias vector $P$. Its action is to cancel the lucky-rate through the identity $\ell = \sum_i p_i q_i = n / \sum_j 1/p_j$, which the paper bounds by $1/n$ using the harmonic mean–arithmetic mean inequality. The secondary mechanism is the Semantic-Spread module, which assigns the most similar distractor to a position sampled with probability $r_j \propto \exp(|i^*-j|)$, so the tempting option is, on average, far from the correct answer. These two rules convert bias measurements into a placement policy: the answer goes where the model does not habitually look, and the closest wrong answer goes where it is easiest to resist.

What would settle it

Measure each model's per-slot selection frequency on real MMLU or CSQA questions whose correct answers are withheld, compare it against the null-prompt distribution $P$, and compute the realized lucky-rate when the answer is placed by inverse-bias sampling; if the empirical selection rate of the correct position on held-out questions exceeds $1/n$, or diverges from $P$ in ranking, the cancellation claim fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a model's positional preference in multiple-choice tests is a measurable, model-specific quantity, and that using its inverse to place answer slots neutralizes the lucky-hit component of accuracy. Section 5.2 states the position-bias cancellation theorem: if $P=(p_1,\ldots,p_n)$ is the position-bias distribution estimated from null prompts and $Q$ is the inverse-bias distribution $q_i = (1/p_i)/\sum_j 1/p_j$, then the lucky-rate $\ell = \sum_i p_i q_i$ is asserted to equal $1/n$; the proof given actually establishes $\ell \le 1/n$, with equality only when $p_i = 1/n$ for all $i$. The corollary is that any measured accuracy above $1/n$ under SCOPE placement must come from content understanding. Together with the Semantic-Spread module, which places the semantically closest distractor at an exponentially distance-weighted slot, the framework claims to yield bias-free evaluations; the experiments report that SCOPE raises Answer F1 on MMLU and CSQA across all eight models tested, while keeping the increase in consistent wrong answers below that of competing debiasing methods.

Load-bearing premise

The load-bearing premise is that the position-bias distribution measured from repeated null prompts is the same as the bias the model applies when answering real content questions; if content changes positional preferences, the inverse placement is miscalibrated and the $1/n$ luck cap does not hold for those items.

Editorial extensions

If this is right

  • Any accuracy above $1/n$ on a SCOPE-arranged test cannot be explained by positional luck, since the lucky-rate is provably no greater than $1/n$.
  • Because the answer slot is re-sampled per question from the inverse-bias distribution, no single position accumulates correct answers, so the evaluation cannot be inflated by an accidental alignment of answers with preferred slots.
  • Separating the most similar distractor by an exponentially distance-weighted draw suppresses near-miss guessing without changing the underlying reasoning requirement, as the adjacency experiments show.
  • The repeated-trial protocol and the Answer/Distractor F1 metric families let an evaluator separate confident correct knowledge (Co-T) from confident misconceptions (Co-F), something plain accuracy hides.
  • SCOPE provides a uniform fairness baseline across models: the lucky-rate bound holds for any position-bias distribution, so models cannot be compared unfairly on the strength of different positional habits.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • [Inference] The whole scheme rests on an unvalidated transfer: the bias measured from meaningless null prompts is assumed to equal each model's positional preference on real, content-rich MMLU and CSQA items; if content changes the ordering of preferences, the reported 'pure skill = F1 − ℓ' is miscalibrated on exactly those items.
  • [Inference] A natural extension is to use SCOPE as a preprocessing layer inside existing calibration pipelines: once positional lucky hits are capped, the remaining confidence signal should be easier to calibrate with methods like temperature scaling, which the paper does not test.
  • [Inference] The framework predicts a testable cross-model consequence: relative rankings between models should be more stable across repeated runs under SCOPE than under random shuffles, because the lucky-rate is held at the same bound for every model.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SCOPE, a two-module framework for multiple-choice evaluation of LLMs. The Inverse-Positioning (IP) module estimates a model's position-bias distribution P by querying it with semantically empty null prompts, then samples the correct-answer slot from the inverse distribution Q. The Semantic-Spread (SS) module identifies the most semantically similar distractor and places it far from the correct answer using an exponential distance weighting. The authors claim a theoretical guarantee that the lucky-rate (probability of a position-based lucky hit) is bounded by 1/n, that SCOPE removes the global concentration of correct answers, and that it yields a 'pure skill' score by subtracting the lucky-rate from Answer F1. Experiments on MMLU and CSQA across eight models report consistent improvements in Answer F1 and a module ablation claiming 63%/37% IP/SS contributions.

Significance. If the core assumptions held, SCOPE would be a useful, dataset-independent debiasing tool for MCQ evaluation. The paper is unusually transparent in shipping a full reproducibility statement with fixed datasets, seeds, and code. The underlying inequality ℓ ≤ 1/n is mathematically correct as an inequality and is a genuine property of the inverse-weighting construction. The empirical scope is large (approximately 700,000 responses, eight models, two benchmarks). However, the theoretical result is misstated as an equality, the proof of the semantic-spread proposition contains an invalid step, the central transfer assumption from null-prompt bias to real-item bias is never validated, and the reported metric definitions are internally inconsistent. These issues are load-bearing for the paper's empirical claims.

major comments (4)
  1. [Section 5.2, Theorem] The theorem is stated as ℓ = 1/n, but the proof establishes only ℓ ≤ 1/n, with equality if and only if P is uniform. For any non-uniform P (e.g., P = (0.8, 0.2) with n = 2), ℓ = 0.32 < 0.5 = 1/n, so the equality statement is false. The corollary's reasoning uses the inequality and is valid, but the theorem as printed is mathematically incorrect. Please restate the theorem as an inequality and adjust any language implying exact equalization of the lucky-rate.
  2. [Sections 4.1 and 5.2, with Table 8] The theoretical bound ℓ = Σ p_i q_i ≤ 1/n holds only when p_i are the model's true selection probabilities on the real test items. The paper measures P using null prompts containing random letter strings and never validates that this equals the position bias on real MMLU/CSQA items. If the true bias P* differs from the null-prompt P, then Σ p*_i q_i can far exceed 1/n; for example, if the real bias is the reverse of the null-prompt bias, the lucky-rate can approach 1. All reported ℓ values (e.g., 0.004 in Table 8) are computed from the null-prompt P itself, not from responses to real items, so the 'pure skill = F1 − ℓ' arithmetic and the 63%/37% IP/SS ablation are not trustworthy. The authors should add an experiment comparing null-prompt selection rates with selection rates on permuted real questions, or otherwise demonstrate that the bound transfers to actual benchmark items.
  3. [Appendix F, Equations (7)–(10)] The proof of the distance-weighted distractor proposition is invalid. The text asserts the chain μ_unif < d_max ≤ μ_R, but the derivation preceding it establishes only μ_R > 1, not μ_R ≥ d_max. For a concrete counterexample to the claimed inequality, take n = 4 with the answer at position 1, so distances are {1, 2, 3} and d_max = 3. Then μ_R = (e^1·1 + e^2·2 + e^3·3)/(e^1 + e^2 + e^3) ≈ 2.575 < 3. The proposition itself may be true, but it requires a correct proof, for instance a direct application of Jensen's inequality to compare μ_R and μ_unif.
  4. [Sections 6.3, 7.2, and Tables 4–6] The definitions of the preference and consistency counts are internally inconsistent. Section 6.3 and Table 4 define Pr-T as 'chose the correct answer at least 3 out of 5 times' and Co-T as 'chose the correct answer in all 5 trials,' which implies Co-T ⊆ Pr-T. However, the raw counts in Table 6 (Claude 3.5-sonnet, CSQA: Pr-T = 82, Co-T = 337) and Table 15 (ChatGPT 3.5-turbo, MMLU: Pr-T = 101, Co-T = 232) violate this subset relation. Section 7.2 instead states that Pr-T and Pr-F are single-run accuracy counts, which is yet another definition. The formulas AR = Co-T/(Pr-T + Co-T) and the resulting F1 values are therefore ambiguous and cannot be reproduced from the stated taxonomy. Please clarify the exact meaning of Pr-T/Pr-F and Co-T/Co-F, correct Table 4, and recompute or re-report all affected metrics consistently.
minor comments (5)
  1. [Section 4.1] The sentence 'since the answer slot is resampled independently for each question, the global concentration of correct answers in specific positions across the evaluation set disappears' is inaccurate. Sampling i* ~ Q produces a marginal distribution equal to Q, which is generally non-uniform; the correct answers are intentionally concentrated in low-bias positions. What disappears is the concentration in high-bias positions, not concentration per se.
  2. [Section 5.4] The statement that Proposition 1 shows the method 'increas[es] evaluation difficulty' conflicts with the paper's central framing that SCOPE removes bias rather than altering test difficulty. This wording should be revised to avoid implying that the evaluation is intentionally made harder.
  3. [Tables 1, 9, 10, 13, 14] The 'Selection Rate' rows in these tables appear to contain placeholder or garbled content (e.g., '15 25 35' and '5 40 75') rather than a clear distribution of selection rates over positions. Please replace these with actual numerical distributions or plots.
  4. [References] Reference [32] (Zheng et al., 'Large language models are not robust multiple choice selectors') duplicates reference [9] with a different year and URL format. Please merge or distinguish these entries.
  5. [Appendix D.1, Algorithm 2] The smoothing formula p_i ← (c_i + ε)/(M + nε) is written ambiguously in the algorithm text, and the main text's claim that 'M ≥ 100, ε ≈ 10^{-3}' is not tied to any sensitivity analysis. Please clarify the formula and provide a short robustness check for ε.

Circularity Check

1 steps flagged · score 5.0 of 10

The headline 'theoretical guarantee' that lucky-rate ≤ 1/n is a definitional consequence of defining Q as the inverse of P; the paper never independently measures the lucky-rate on real items, so the pure-skill correction inherits the method's own estimate.

  1. self definitional [Section 4.1 (Inverse-Positioning module) and Section 5.2 (Position-bias cancellation theorem and corollary)]
    "Then, the answer slot is determined based on probabilities proportional to the inverse of P . For each position i, we define: qi = 1Pn j=1 1/pj , Q = (q1, . . . , qn) ... First, it effectively eliminates the lucky hit by bounding the lucky-rate ℓ—i.e., the probability of guessing the correct answer based solely on positional cues—to at most 1/n. ... ℓ = Pn i=1 piqi ... which implies ℓ ≤ n n2 = 1 n."

    Q is defined as the normalized inverse of P, and ℓ is defined as Σ p_i q_i. Substituting the definition gives ℓ = n / Σ 1/p_j, so the bound ℓ ≤ 1/n follows from HM-AM for any probability vector P regardless of model behavior on actual questions. The corollary's 'guarantee' that accuracy above 1/n must come from understanding therefore restates the construction, and it holds only if the null-prompt P equals the model's true position bias on real items. The paper never measures the realized lucky-rate on real MCQs, so the theorem is a definitional property of the inverse-bias construction rather than an independent empirical finding.

full rationale

The empirical benchmark comparisons are self-contained: SCOPE is tested against external baselines on MMLU and CSQA, and the paper provides code and fixed seeds for reproduction, so the performance claims do not reduce to the method's own inputs. However, the paper's flagship 'theoretical guarantee' in Section 5.2 is a definitional artifact. Q is defined as the inverse of the null-prompt bias P, and the lucky-rate ℓ is defined as Σ p_i q_i; the HM-AM bound ℓ ≤ 1/n is an algebraic identity of that definition. The corollary that any accuracy above 1/n is guaranteed understanding is therefore only as strong as the untested assumption that P measured from meaningless prompts equals the position bias on real MMLU/CSQA items. All reported ℓ values (e.g., 0.004 in Table 8) and the resulting 'pure skill = F1 − ℓ' numbers are computed from this assumed P rather than from an independent measurement, so the ablation decomposition (63%/37%) inherits the same unvalidated transfer. This is a partial, definition-level circularity in the theoretical claim, not in the benchmark comparisons. There is no load-bearing self-citation chain, and the mathematical inequality itself is correct; the issue is that the paper presents a property of its own construction as an externally meaningful guarantee.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central claim depends on the transferability of null-prompt bias to real questions, the additivity of bias and knowledge, and the reliability of embedding-based distractor ranking. These are domain assumptions that are not validated in the paper. The method also introduces several hand-chosen constants (M=1000, ε≈1e-3, exponential distance weighting, 5 repetitions, 3/5 and 5/5 thresholds).

free parameters (6)
  • Null-prompt template = You must choose one. If you had to pick, which would it be?
    The position bias distribution P is estimated with this single template; the paper gives no evidence that the bias measured with this template matches bias on real MCQs.
  • Null-prompt budget M = 1000
    Set by hand; no sensitivity analysis.
  • Laplace smoothing epsilon = ~1e-3 (Appendix D)
    Chosen by hand, stated as generally sufficient.
  • Distance weight function = exp(d_j) in main text, power t^tau in Appendix D
    The choice of exponential weighting affects the SSD placement distribution; no ablation over tau.
  • Repetitions per question = 5
    Chosen as 'statistically minimal' with no formal justification.
  • Preference and consistency thresholds = 3/5 and 5/5
    Arbitrary cutoffs for defining Pr and Co metrics.
assumptions (6)
  • standard math Harmonic-arithmetic mean inequality (HM ≤ AM) for positive reals
    Used in Section 5.2 to derive ℓ ≤ 1/n.
  • standard math Jensen's inequality for convex functions
    Invoked in Section 5.3 and Appendix F for the distance-dispersion proposition.
  • domain assumption Null-prompt position bias equals real-question position bias
    Section 4.1 uses P from nonsense prompts to construct Q for real datasets.
  • domain assumption Position bias and content understanding are independent and additive
    Required for the lucky-rate decomposition ℓ = Σ p_i q_i and the pure-skill subtraction F1 - ℓ.
  • domain assumption Sentence-BERT cosine similarity ranks distractor confusability
    Section 4.2 identifies the SSD using all-MiniLM-L6-v2 embeddings.
  • domain assumption Five repeated responses with temperature 1 produce stable knowledge signals
    Section 6.1 fixes 5 repetitions and uses thresholds 3/5 and 5/5 for Pr and Co.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SCOPE: Stochastic and Counterbiased Option Placement for Evaluating Large Language Models." pith.science (2026). https://pith.science/paper/PNVWP2QJ

@misc{pith2026250718182,
  author       = {Pith},
  title        = {Pith review of: SCOPE: Stochastic and Counterbiased Option Placement for Evaluating Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PNVWP2QJ}},
  note         = {Machine review of arXiv:2507.18182}
}
read the original abstract

Large Language Models (LLMs) can achieve inflated scores on multiple-choice tasks by exploiting inherent biases in option positions or labels, rather than demonstrating genuine understanding. This study introduces SCOPE, an evaluation framework designed to measure and mitigate such selection bias in a dataset-independent manner. By repeatedly invoking a null prompt that lacks semantic content, SCOPE estimates each model's unique position-bias distribution. It then redistributes the answer slot according to the inverse-bias distribution, thereby equalizing the lucky-rate, the probability of selecting the correct answer by chance. Furthermore, it prevents semantically similar distractors from being placed adjacent to the answer, thereby blocking near-miss guesses based on superficial proximity cues. Across multiple benchmark experiments, SCOPE consistently outperformed existing debiasing methods in terms of stable performance improvements and showed clearer confidence distributions over correct options. This framework thus offers a new standard for enhancing the fairness and reliability of LLM evaluations.

Figures

Figures reproduced from arXiv: 2507.18182 by the authors.

Figure 1
Figure 1. Overall architecture of the SCOPE framework (IP + SS) [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Rethinking Query Optimization for Multi-Agent Systems [Vision]

    cs.DB 2025-12 conditional novelty 6.0 of 10

    Agentic data pipelines are built by hand today; this paper sets a research agenda for automatically optimizing their structure, model choices, and execution engines jointly as a new query-optimization problem.

Reference graph

Works this paper leans on

78 extracted references · 44 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Sparks of artificial general intelligence: Early experiments with gpt-4

    Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023. 15 Jeong et al

    arXiv 2023

  3. [3]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  4. [4]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    arXiv 2023

  5. [5]

    The economic potential of generative ai: The next productivity frontier, 2023

    McKinsey Global Institute. The economic potential of generative ai: The next productivity frontier, 2023

    work page 2023

  6. [6]

    Pwc is accelerating adoption of ai with chatgpt enterprise in us and uk and with clients, 2024

    PwC. Pwc is accelerating adoption of ai with chatgpt enterprise in us and uk and with clients, 2024

    work page 2024

  7. [7]

    Beyond accuracy: evaluating the reasoning behavior of large language models–a survey

    Philipp Mondorf and Barbara Plank. Beyond accuracy: evaluating the reasoning behavior of large language models–a survey. arXiv preprint arXiv:2404.01869, 2024

    arXiv 2024

  8. [8]

    Shortcut learning of large language models in natural language understanding

    Mengnan Du, Fengxiang He, Na Zou, Dacheng Tao, and Xia Hu. Shortcut learning of large language models in natural language understanding. Commun. ACM, 67(1):110–120, December 2023

    work page 2023

Show all 78 references
  1. [10]

    Anchored answers: Unravelling positional bias in gpt-2’s multiple-choice questions

    Ruizhe Li and Yanjun Gao. Anchored answers: Unravelling positional bias in gpt-2’s multiple-choice questions. arXiv preprint arXiv:2405.03205, 2024

    2024 arXiv

  2. [11]

    Calibraeval: Calibrating prediction distribution to mitigate selection bias in llms-as-judges

    Haitao Li, Junjie Chen, Qingyao Ai, Zhumin Chu, Yujia Zhou, Qian Dong, and Yiqun Liu. Calibraeval: Calibrating prediction distribution to mitigate selection bias in llms-as-judges. arXiv preprint arXiv:2410.15393, 2024

    2024 arXiv

  3. [12]

    Look at the text: Instruction-tuned language models are more robust multiple choice selectors than you think

    Xinpeng Wang, Chengzhi Hu, Bolei Ma, Paul Röttger, and Barbara Plank. Look at the text: Instruction-tuned language models are more robust multiple choice selectors than you think. arXiv preprint arXiv:2404.08382, 2024

    2024 arXiv

  4. [13]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  5. [14]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020

    2020

  6. [15]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  7. [16]

    Commonsenseqa: A question answering challenge targeting commonsense knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua...

    2019

  8. [17]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  9. [18]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, 2018

    2018

  10. [19]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on mach...

    2023

  11. [20]

    Holistic evaluation of language models

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022

    2022 arXiv

  12. [21]

    M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models

    Wenxuan Zhang, Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models. Advances in Neural Information Processing Systems, 36:5484–5505, 2023

    2023

  13. [22]

    Agieval: A human-centric benchmark for evaluating foundation models

    Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2299–2314, 202...

    2024

  14. [23]

    Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pages 248–260. PMLR, 2022

    2022

  15. [24]

    Crowdsourcing multiple choice science questions

    Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 94–106, 2017

    2017

  16. [25]

    From live data to high-quality benchmarks: The arena-hard pipeline

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From live data to high-quality benchmarks: The arena-hard pipeline. Blog post.[Accessed 07-02-2025], 2024

    2025

  17. [26]

    Chatbot arena: An open platform for evaluating llms by human preference

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. In Forty-first International Conference...

    2024

  18. [27]

    Large language models are not fair evaluators

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, et al. Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa...

    2024

  19. [28]

    Apbench and benchmarking large language model performance in fundamental astrodynamics problems for space engineering

    Di Wu, Raymond Zhang, Enrico M Zucchelli, Yongchao Chen, and Richard Linares. Apbench and benchmarking large language model performance in fundamental astrodynamics problems for space engineering. Scientific Reports, 15(1):7944, 2025

    2025

  20. [29]

    Where is the answer? an empirical study of positional bias for parametric knowledge extraction in language model

    Kuniaki Saito, Chen-Yu Lee, Kihyuk Sohn, and Yoshitaka Ushiku. Where is the answer? an empirical study of positional bias for parametric knowledge extraction in language model. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for ...

    2025

  21. [30]

    Option symbol matters: Investigating and mitigating multiple-choice option symbol bias of large language models

    Zhen Yang, Ping Jian, and Chengzhi Li. Option symbol matters: Investigating and mitigating multiple-choice option symbol bias of large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics...

    2025

  22. [31]

    Large language models sensitivity to the order of options in multiple- choice questions

    Pouya Pezeshkpour and Estevam Hruschka. Large language models sensitivity to the order of options in multiple- choice questions. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2006–2017, 2024

    2024

  23. [32]

    Large language models are not robust multiple choice selectors

    Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors. URL https://arxiv. org/abs/2309.03882, 2024

    2024 arXiv

  24. [33]

    Fool your (vision and) language model with embarrassingly simple permutations

    Yongshuo Zong, Tingyang Yu, Ruchika Chavhan, Bingchen Zhao, and Timothy Hospedales. Fool your (vision and) language model with embarrassingly simple permutations. In International Conference on Machine Learning, pages 62892–62913. PMLR, 2024

    2024

  25. [34]

    Teacher-student training for debiasing: General permutation debiasing for large language models

    Adian Liusie, Yassir Fathullah, and Mark Gales. Teacher-student training for debiasing: General permutation debiasing for large language models. In Findings of the Association for Computational Linguistics ACL 2024, pages 1376–1387, 2024

    2024

  26. [35]

    Mitigating selection bias with node pruning and auxiliary options

    Hyeong Kyu Choi, Weijie Xu, Chi Xue, Stephanie Eckman, and Chandan K Reddy. Mitigating selection bias with node pruning and auxiliary options. arXiv preprint arXiv:2409.18857, 2024

    2024 arXiv

  27. [36]

    Unveiling selection biases: Exploring order and token sensitivity in large language models

    Sheng-Lun Wei, Cheng-Kuang Wu, Hen-Hsen Huang, and Hsin-Hsi Chen. Unveiling selection biases: Exploring order and token sensitivity in large language models. In Findings of the Association for Computational Linguistics ACL 2024, pages 5598–5621, 2024

    2024

  28. [37]

    Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  29. [38]

    Large language models are zero-shot reasoners

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213, 2022

    2022

  30. [39]

    Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2022

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2022

    2022 arXiv

  31. [40]

    Star: Self-taught reasoner bootstrapping reasoning with reasoning

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D Goodman. Star: Self-taught reasoner bootstrapping reasoning with reasoning. In Proc. the 36th International Conference on Neural Information Processing Systems, volume 1126, 2024

    2024

  32. [41]

    Least-to-most prompting enables complex reasoning in large language models

    Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625, 2022. 17 Jeong et al

    2022 arXiv

  33. [42]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023

    2023

  34. [43]

    Debiasing in-context learning by instructing llms how to follow demonstrations

    Lvxue Li, Jiaqi Chen, Xinyu Lu, Yaojie Lu, Hongyu Lin, Shuheng Zhou, Huijia Zhu, Weiqiang Wang, Zhongyi Liu, Xianpei Han, et al. Debiasing in-context learning by instructing llms how to follow demonstrations. In Findings of the Association for Computational Linguistics ACL 202...

    2024

  35. [44]

    Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks

    Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588, 2022

    2022 arXiv

  36. [45]

    Prompt sketching for large language models

    Luca Beurer-Kellner, Mark Niklas Mueller, Marc Fischer, and Martin Vechev. Prompt sketching for large language models. In Proceedings of the 41st International Conference on Machine Learning, pages 3674–3706, 2024

    2024

  37. [46]

    Self-refine: Iterative refinement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534–46594, 2023

    2023

  38. [47]

    Constitutional ai: Harmlessness from ai feedback

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022

    2022 arXiv

  39. [48]

    Neurologic decoding:(un) supervised neural text generation with predicate logic constraints

    Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Neurologic decoding:(un) supervised neural text generation with predicate logic constraints. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Com...

    2021

  40. [49]

    Calibration of pre-trained transformers

    Shrey Desai and Greg Durrett. Calibration of pre-trained transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 295–302, 2020

    2020

  41. [50]

    Calibrate before use: Improving few-shot performance of language models

    Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International conference on machine learning, pages 12697–12706. PMLR, 2021

    2021

  42. [51]

    Calibrating language models with adaptive temperature scaling

    Johnathan Xie, Annie Chen, Yoonho Lee, Eric Mitchell, and Chelsea Finn. Calibrating language models with adaptive temperature scaling. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18128–18138, 2024

    2024

  43. [52]

    Calibrating large language models with sample consistency

    Qing Lyu, Kumar Shridhar, Chaitanya Malaviya, Li Zhang, Yanai Elazar, Niket Tandon, Marianna Apidianaki, Mrinmaya Sachan, and Chris Callison-Burch. Calibrating large language models with sample consistency. In Proceedings of the AAAI Conference on Artificial Intelligence, volu...

    2025

  44. [53]

    Benchmarking uncertainty quantification methods for large language models with lm-polygraph

    Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing, Abdelrahman Sadallah, Kirill Grishchenkov, et al. Benchmarking uncertainty quantification methods for large language models with lm-polygraph. Transa...

    2025

  45. [54]

    Thermometer: towards universal calibration for large language models

    Maohao Shen, Subhro Das, Kristjan Greenewald, Prasanna Sattigeri, Gregory Wornell, and Soumya Ghosh. Thermometer: towards universal calibration for large language models. In Proceedings of the 41st International Conference on Machine Learning, pages 44687–44711, 2024

    2024

  46. [55]

    Monte carlo temperature: a robust sampling strategy for llm’s uncertainty quantification methods

    Nicola Cecere, Andrea Bacciu, Ignacio Fernández-Tobías, and Amin Mantrach. Monte carlo temperature: a robust sampling strategy for llm’s uncertainty quantification methods. In Proceedings of the 5th Workshop on Trustworthy NLP (TrustNLP 2025), pages 305–320, 2025

    2025

  47. [56]

    Charm: Calibrating reward models with chatbot arena scores

    Xiao Zhu, Chenmien Tan, Pinzhen Chen, Rico Sennrich, Yanlin Zhang, and Hanxu Hu. Charm: Calibrating reward models with chatbot arena scores. arXiv preprint arXiv:2504.10045, 2025

    2025

  48. [57]

    Restoring calibration for aligned large language models: A calibration-aware fine-tuning approach

    Jiancong Xiao, Bojian Hou, Zhanliang Wang, Ruochen Jin, Qi Long, Weijie J Su, and Li Shen. Restoring calibration for aligned large language models: A calibration-aware fine-tuning approach. arXiv preprint arXiv:2505.01997, 2025

    2025

  49. [58]

    Uncertainty estimation in large language models to support biodiversity conservation

    Maria Mora-Cross and Saul Calderon-Ramirez. Uncertainty estimation in large language models to support biodiversity conservation. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (V...

    2024

  50. [59]

    Theory of mind may have spontaneously emerged in large language models

    Michal Kosinski. Theory of mind may have spontaneously emerged in large language models. arXiv preprint arXiv:2302.02083, 4:169, 2023

    2023 arXiv

  51. [60]

    Neural theory-of-mind? on the limits of social intelligence in large lms

    Maarten Sap, Ronan Le Bras, Daniel Fried, and Yejin Choi. Neural theory-of-mind? on the limits of social intelligence in large lms. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3762–3780, 2022. 18 Jeong et al

    2022

  52. [61]

    Social iqa: Commonsense reasoning about social interactions

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social iqa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural ...

    2019

  53. [62]

    Large language models are not strong abstract reasoners

    Gaël Gendron, Qiming Bao, Michael Witbrock, and Gillian Dobbie. Large language models are not strong abstract reasoners. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, pages 6270–6278, 2024

    2024

  54. [63]

    Coglm: Tracking cognitive development of large language models

    Xinglin Wang, Peiwen Yuan, Shaoxiong Feng, Yiwei Li, Boyuan Pan, Heda Wang, Yao Hu, and Kan Li. Coglm: Tracking cognitive development of large language models. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Ling...

    2025

  55. [64]

    V-alphasocial: Benchmark and self-reflective chain-of- thought generation for visual social commonsense reasoning

    Zongyu Lin, Zhikun Xu, Xiaohan Song, Yixin Wan, Xingcheng Yao, Tsung-Han Lin, Selina Song, Pranav Subbaraman, Ben Zhou, Kai-Wei Chang, and Yizhou Sun. V-alphasocial: Benchmark and self-reflective chain-of- thought generation for visual social commonsense reasoning. In ACL-Find...

    2025

  56. [65]

    Mind2web: Towards a generalist agent for the web

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091–28114, 2023

    2023

  57. [66]

    Mind2web 2: Evaluating agentic search with agent-as-a-judge

    Boyu Gou, Zanming Huang, Yuting Ning, Yu Gu, Michael Lin, Weijian Qi, Andrei Kopanev, Botao Yu, Bernal Jiménez Gutiérrez, Yiheng Shu, et al. Mind2web 2: Evaluating agentic search with agent-as-a-judge. arXiv preprint arXiv:2506.21506, 2025

    2025 arXiv

  58. [67]

    Long range arena: A benchmark for efficient transformers

    Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04006, 2020

    2011 arXiv

  59. [68]

    Minerva: A programmable memory test benchmark for language models

    Menglin Xia, Victor Ruehle, Saravan Rajmohan, and Reza Shokri. Minerva: A programmable memory test benchmark for language models. arXiv preprint arXiv:2502.03358, 2025

    2025 arXiv

  60. [69]

    L-eval: Instituting standardized evaluation for long context language models

    Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vo...

    2024

  61. [70]

    Needle in the haystack for memory based large language models

    Elliot Nelson, Georgios Kollias, Payel Das, Subhajit Chaudhury, and Soham Dan. Needle in the haystack for memory based large language models. arXiv preprint arXiv:2407.01437, 2024

    2024 arXiv

  62. [71]

    Con- versational ai powered by large language models amplifies false memories in witness interviews

    Samantha Chan, Pat Pataranutaporn, Aditya Suri, Wazeer Zulfikar, Pattie Maes, and Elizabeth F Loftus. Con- versational ai powered by large language models amplifies false memories in witness interviews. arXiv preprint arXiv:2408.04681, 2024

    2024 arXiv

  63. [72]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)...

    2019

  64. [73]

    Introduction to information retrieval , volume 39

    Hinrich Schütze, Christopher D Manning, and Prabhakar Raghavan. Introduction to information retrieval , volume 39. Cambridge University Press Cambridge, 2008

    2008

  65. [74]

    The claude 3 model family: Opus, sonnet, haiku

    Anthropic. The claude 3 model family: Opus, sonnet, haiku. https://www.anthropic.com/ claude-3-model-card , 2024

    2024

  66. [75]

    Model card addendum: Claude 3.5 haiku and sonnet

    Anthropic. Model card addendum: Claude 3.5 haiku and sonnet. https://assets.anthropic.com/.../ Claude-3-5-Addendum.pdf , 2025

    2025

  67. [76]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  68. [77]

    Introducing meta llama 3

    Meta AI. Introducing meta llama 3. https://ai.meta.com/blog/meta-llama-3/ , April 2024

    2024

  69. [78]

    The serial position effect of free recall

    Bennet B Murdock Jr. The serial position effect of free recall. Journal of experimental psychology, 64(5):482, 1962

    1962

  70. [79]

    luck-free

    Murray Glanzer and Anita R Cunitz. Two storage mechanisms in free recall. Journal of verbal learning and verbal behavior, 5(4):351–360, 1966. 19 Jeong et al. Appendix A Selection distribution across evaluation conditions Table 9 presents a quantitative analysis of how fixed an...

    1966

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.