Pith. sign in

REVIEW 4 major objections 6 minor 66 references

Every Wrong Answer Counts: Option-Level Psychometrics for LLM Multiple-Choice Benchmarks

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An option-aware psychometric model that scores which wrong answer an LLM picks predicts unseen responses better and agrees more closely with human preference rankings, according to this paper.

desk verdict Solid option-level IRT extension for LLM evaluation with real held-out gains, but the claim that wrong answers carry independent ability signal still needs an external anchor. read the letter →

arxiv 2608.02966 v1 pith:6ACCDOT3 submitted 2026-08-03 cs.CL

Xiao Fei , Yang Zhang , Sarah Almeida Carneiro , Michalis Vazirgiannis This is my paper · ORCID
classification cs.CL
keywords LLMevaluationitemresponsetheorynominalmodelmultiple-choicebenchmarksdistractoranalysispsychometricsabilityestimationbenchmarkcompression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multiple-choice benchmarks for LLMs score a response as right or wrong, throwing away which wrong option was chosen. This paper argues that the wrong answers are not equivalent errors, and that the full probability distribution an LLM places over the options is itself a measurement signal. It introduces LLM-NRM, an option-aware psychometric model of latent ability, item characteristics, and LLM-specific response behaviors. Across 189 LLMs and 31,554 items, the model predicts held-out responses better than binary item response models, and its ability estimates line up more closely with an external human-preference leaderboard. If correct, the result turns distractor choices from noise into data: benchmarking could be compressed hundreds of times and rank models without ever observing a correct answer.

What carries the argument

The central object is LLM-NRM, an extension of Bock's nominal response model in which each answer option has its own discrimination and intercept parameters, and the respondent model adds a per-LLM inverse-temperature sharpness, a positional-bias vector, and a difficulty-gated guessing fallback. The fallback gate is a logistic function of the gap between LLM ability and a differentiable item-difficulty index derived from the NRM option parameters, and it interpolates between ability-driven utilities and a per-LLM fallback distribution over positions. This joint model separates calibration sharpness, content-independent position preference, and difficulty-dependent guessing from ability and item content, which the paper argues prevents these LLM-specific behaviors from contaminating ability estimates.

What would settle it

Take a set of MCQ items in which the correct option and at least one distractor share the same first token, recompute observed option probabilities from the full token sequence rather than first-token mass, and compare LLM-NRM's held-out log-loss against the first-token-based estimates; if the gap shrinks or reverses, the first-token proxy is the load-bearing simplification.

Watch

Extended reading notes

Core claim

The paper establishes that the full categorical distribution an LLM assigns to multiple-choice options, not just the selected answer, behaves as a psychometric response. Modeling this distribution with an adapted nominal response model yields better held-out response prediction than binary IRT and conventional nominal baselines, and the resulting ability estimates agree more closely with external human-preference Elo rankings than raw accuracy or any of the competing latent-ability estimates. Distractor identity alone is shown to contribute about twice as much Fisher information per item as correctness, and incorrect responses alone recover full-information ability estimates with Spearman correlation around 0.943.

Load-bearing premise

The entire model assumes that the observed option distribution, first-token log-probability mass on each answer option with zero for options missing from returned top-20 logits, faithfully represents the model's true preference among complete answers.

Editorial extensions

If this is right

  • Benchmark designers can use option-level item parameters to identify which distractors actually discriminate ability and to write more informative wrong answers.
  • A new LLM can be ranked reliably from as few as 41 selected items, a 770-fold reduction in evaluation cost.
  • Because error patterns alone recover ability at Spearman 0.943, settings where correct answers are hidden or partially observed still permit meaningful model measurement.
  • Separating positional bias and calibration sharpness from ability lets reported accuracy gaps be decomposed into knowledge differences versus response-style differences.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The Fisher information gain should persist if option probabilities are recomputed from full-sequence scores rather than first-token mass; testing this would reveal how much of the reported advantage rests on the first-token proxy.
  • The strong agreement with external human-preference Elo suggests option-level modeling could serve as a cheap proxy for pairwise human evaluation, which could be validated by predicting individual pairwise judgments rather than aggregate scores.
  • The finding that newer benchmarks contain more informative distractors hints that distractor construction is a rising design axis, potentially guiding automatic distractor generation by maximizing Fisher information.
  • The single latent dimension is a deliberate simplification; a multidimensional extension could separate factual knowledge from reasoning and might also detect prompt-order contamination through the estimated positional parameters.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript introduces LLM-NRM, a nominal-response item response theory model that treats an LLM's full probability distribution over MCQ answer options as the psychometric response. The model augments Bock's NRM with per-LLM response sharpness, positional bias, and a difficulty-gated fallback process, and is fit by MAP estimation on soft cross-entropy to 189 LLMs and 31,554 items from 14 benchmarks. The central empirical claims are: (i) LLM-NRM predicts held-out LLM-item responses better than binary IRT and NRM baselines, especially in the option channel; (ii) its ability estimates correlate more strongly with the external Arena.ai Text Elo leaderboard (Spearman 0.920) than raw accuracy or baseline IRT abilities; (iii) distractor identity adds +101% Fisher information beyond binary correctness; (iv) ability estimated from incorrect responses alone recovers the full-information ability estimate with Spearman 0.943; and (v) 41 information-selected items preserve the full-bank ranking with Kendall's tau 0.85.

Significance. The paper addresses a real and timely problem: binary accuracy scoring discards option-level response structure in LLM MCQ evaluation. The scale of the study, the 5-fold cell-level cross-validation, the comparison against a wide set of IRT and deep-learning baselines, and the independent Arena Elo check for full-response ability are substantial strengths. If the option-level information claims survive closer scrutiny, the model would be a useful tool for benchmark analysis, ability estimation, and test compression. The main gap is that the specific claim that incorrect answers carry distinct and useful measurement information is not yet anchored to an external criterion: the held-out prediction and Elo validation are for the full response, while the incorrect-only evidence is internal to the fitted model. The paper is suitable for publication after these validation gaps are addressed.

major comments (4)
  1. [Answer-choice distributions] The observed response p_obs_ji is defined as the first-token log-probability mass on each valid option, with options absent from top-20 API logits assigned zero mass. This measurement proxy is load-bearing: every fitted curve, Fisher-information computation, and compression result inherits any error in this construction. Options that share first tokens, for example 'the Sun' versus 'the Moon', are not separated by first-token probability mass, and top-20 truncation can drop probability mass that is relevant to the full option distribution. Please report the fraction of items with first-token collisions and provide a robustness check on a subset where full option probabilities are obtained by forced decoding, or state explicitly the conditions under which the first-token proxy is valid.
  2. [Ability Estimation from Incorrect Responses, Table 3] The incorrect-only ability estimates are validated only against the full-information theta from the same fitted model. The external Arena Elo validation in Table 2 uses full responses and therefore does not test the wrong-answer claim. Because the two estimates in Table 3 share item parameters and the same likelihood, their Spearman 0.943 could partly reflect parameter coupling rather than genuinely independent distractor signal. Please add an external anchor for incorrect-only ability, for example the Spearman correlation of incorrect-only theta with Arena Elo on the matched LLMs, or a held-out prediction experiment in which only incorrect responses are used to predict correctness on new items or new LLMs.
  3. [Fisher Information from Distractor Identity, Eq. (16), Figure 2(a)] The decomposition I_NRM = I_binary + I_wrong with I_wrong >= 0 is a mathematical identity for any nominal-response model with option-specific discriminations; the +101% figure is therefore a property of the fitted parameters, not an independent empirical measurement about LLMs. Please present this quantity as a model-based result with uncertainty, for example by bootstrapping over fitted parameters, and pair it with a behavioral outcome such as the option-channel held-out gain in Table 1 to support the claim that wrong-answer identity is empirically useful.
  4. [Compression of MCQ Benchmarks, Figure 3] The text does not make explicit whether the item parameters and the information-based item selection exclude the target held-out LLM. If the item bank and the selection criterion are calibrated on the full fleet including the evaluated LLM, the reported 41-item Kendall tau of 0.85 is optimistic. Please specify the exact split: item calibration on training LLMs only, item selection on training LLMs only, and evaluation on held-out LLMs.
minor comments (6)
  1. [Abstract and Table 3] The abstract reports Spearman 0.943 while Table 3 reports 0.9428 +/- 0.0174; please align these numbers.
  2. [Table 1] The distinction between the soft-channel and hard-channel NRM baselines is not defined before the table is presented; please add one sentence describing what each channel means.
  3. [Figure 2(b)] The correlation between informative-distractor count and benchmark publication date is based on only 14 points; the Pearson r = 0.87 should be interpreted cautiously and the analysis would benefit from a confidence interval.
  4. [Experimental Setup] The descriptions of prior baselines PSN-IRT and SD-IR are brief; please include enough detail to make clear how they were adapted to the option-response setting.
  5. [Fitting] The sentence stating that LLM-NRM estimates 2(K_i - 1) free parameters per MCQ item should clarify that this counts discriminations and intercepts after imposing the sum-to-zero identification constraints.
  6. [Discussion and Conclusion] The limitation paragraph mentions partial probability observations and multidimensional abilities as future work, but does not discuss the sensitivity of results to the choice of the N(0,1) ability prior; a brief comment on prior sensitivity would strengthen the paper.

Circularity Check

1 steps flagged · score 5.0 of 10

Incorrect-only ability recovery is contaminated by item parameters fitted on full responses; central claim retains independent held-out and Arena Elo support.

  1. fitted input called prediction [Ability Estimation from Incorrect Responses (Table 3 and surrounding text)]
    "In each fold, MCQ item parameters are fit using the training LLMs and then held fixed while estimating θ for the held-out LLMs under two conditions: (i) Binary-Only, where the estimator observes only correctness signals; and (ii) Incorrect-Only, where it observes only the categorical signals from incorrectly answered MCQ items. For each held-out LLM, we compare the estimated θ with the reference estimate obtained using the full response information."

    The item parameters used in the Incorrect-Only condition are estimated from training LLMs' full responses, including correct/incorrect outcomes. The mapping from distractor choices to ability is therefore learned with access to correctness information. The reference θ is also computed from the same fitted item parameters and the same likelihood. The high Spearman correlation (0.9428) between Incorrect-Only and full-information θ can thus reflect parameter sharing and model self-consistency rather than an independent demonstration that incorrect responses alone carry ability information. The external Arena Elo validation (Table 2) is never applied to Incorrect-Only estimates, so the paper's headline claim that wrong answers alone recover ability is not anchored to an outside criterion.

full rationale

The core derivation is not circular: LLM-NRM is fitted to observed option distributions, and its held-out response prediction (Table 1) is a genuine out-of-sample comparison against binary IRT and NRM baselines. The Arena Elo validation (Table 2) is an external, independent criterion not used in training. These give the central claim real empirical content. The main circularity concern is the Incorrect-Only experiment: item parameters are calibrated on full responses (including correctness) before being used to estimate ability from incorrect responses only, so the experiment does not isolate information contained in wrong answers. The +101% Fisher information gain is a model-derived quantity from the standard NRM decomposition, not an independent measurement; it is illustrative rather than load-bearing. No load-bearing self-citation chain or uniqueness-theorem argument appears. Score 5 reflects one partially circular supporting experiment while the central claim retains independent support.

Assumptions & free parameters 7 free parameters · 6 assumptions · 3 invented entities

The model rests on a large set of fitted parameters: item-level option curves, per-LLM ability, sharpness, positional bias, and fallback logits, plus two global gate parameters. The central psychometric assumptions are local independence, unidimensional ability, faithful extraction of option probabilities from first-token logits, and the validity of Arena Elo as an external criterion. The model also introduces three latent response constructs (sharpness, positional bias, fallback) that are estimated from the same data and lack independent falsifiable handles.

free parameters (7)
  • Item option discriminations a_ik = MAP estimates for 31,554 items, 2(K_i-1) per item
    Core NRM parameters controlling how each option's attractiveness varies with ability; estimated jointly from the response matrix.
  • Item option intercepts c_ik = MAP estimates, same scale
    Baseline attractiveness of each option; needed for the softmax utilities.
  • LLM ability theta_j = MAP with N(0,1) prior for 189 LLMs
    Target latent variable; identified by the prior and item parameters.
  • Per-LLM sharpness s_j = MAP, softplus reparameterization
    Inverse temperature separating calibration from knowledge; no external anchor.
  • Per-LLM positional bias delta_jk = MAP, sum-to-zero across positions
    Content-independent position preference estimated jointly with ability.
  • Per-LLM fallback logits rho_jk = MAP, sum-to-zero
    Response pattern used in the difficulty-gated fallback regime.
  • Global fallback parameters (w0, kappa) = MAP scalars
    Location and slope of the logistic gate G_ji.
assumptions (6)
  • domain assumption Local independence: responses are conditionally independent given theta_j and item parameters.
    Standard IRT assumption, stated implicitly by the softmax factorization in Eq (13).
  • domain assumption The first-token option probabilities p_obs faithfully represent LLM preferences over full options; unreturned API logits are zero.
    Section 'Answer-choice distributions'; load-bearing for all measurements in the paper.
  • domain assumption A single latent ability dimension theta suffices to explain LLM MCQ responses.
    Acknowledged in the conclusion as a current limitation, but used throughout the model.
  • ad hoc to paper The difficulty index tilde_b_i computed by local linearization of f_i around theta=0 is a valid proxy for item difficulty.
    Eq (9)-(10) use a one-step Newton approximation with no error analysis; this index feeds the fallback gate in Eq (7).
  • domain assumption Arena.ai Elo scores are a valid external criterion for LLM ability.
    Used as ground truth for external validity in Table 2; derived from crowdsourced pairwise preferences on a different prompt type.
  • standard math Softmax with sum-to-zero constraints identifies all model parameters.
    Identification constraints in Eq (3) and (14) are imposed, but no formal identifiability proof is given for s_j, delta_j, and rho_j jointly.
invented entities (3)
  • Per-LLM response sharpness s_j
    purpose: Separate calibration confidence from knowledge in the response distribution
    Estimated from the same response matrix; no external calibration measure is provided.
  • Per-LLM positional bias delta_j
    purpose: Capture content-independent preference for answer positions
    No permutation experiment is provided to validate the estimated positional effects; estimated jointly with ability.
  • Per-LLM fallback vector rho_j and gate G_ji
    purpose: Model difficulty-dependent guessing behavior
    No independent behavioral observation; the gate uses a locally linearized difficulty index from fitted item parameters.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Every Wrong Answer Counts: Option-Level Psychometrics for LLM Multiple-Choice Benchmarks." pith.science (2026). https://pith.science/paper/6ACCDOT3

@misc{pith2026260802966,
  author       = {Pith},
  title        = {Pith review of: Every Wrong Answer Counts: Option-Level Psychometrics for LLM Multiple-Choice Benchmarks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6ACCDOT3}},
  note         = {Machine review of arXiv:2608.02966}
}
read the original abstract

Most multiple-choice question (MCQ) benchmarks evaluate Large Language Models (LLMs) only by whether they select the correct answers. This binary scoring treats all incorrect responses alike, even though an LLM's preferences among incorrect options may contain systematic and useful information about its behavior and ability. We introduce the LLM Nominal Response Model (LLM-NRM), an option-aware psychometric framework that models the full distribution over answer choices to jointly estimate LLM ability and option-level item characteristics, while separating model-specific response calibration sharpness, positional preference, and difficulty-dependent fallback behavior. Across 189 LLMs and 31,554 items from 14 benchmarks, LLM-NRM predicts held-out LLM-item interactions more accurately than binary Item Response models and conventional nominal-response baselines, and its ability estimates achieve the strongest Spearman correlation of 0.920 with the external human-preference Arena.ai Elo leaderboard. Distractor identity contributes +101% additional Fisher Information per item beyond correctness, and incorrect responses alone recover full-information ability estimates with Spearman 0.943. The learned item parameters also enable efficient benchmarking, where 41 selected items preserve the full-bank ranking with Kendall's correlation 0.85, corresponding to a 770 times reduction. In conclusion, we show that incorrect answers carry distinct and useful measurement information rather than representing equivalent mistakes.

Figures

Figures reproduced from arXiv: 2608.02966 by the authors.

Figure 1
Figure 1. A real example of an ARC-Challenge MCQ item with item characteristic curves from our LLM-NRM fitting. (a) In this item, "Saturn" is a plausible distractor that reflects partial knowledge of LLM examinees, whereas "the Sun" is the correct answer. (b) Binary scoring collapses all three choices into the same "incorrect" outcome, but option-level modeling estimates a separate response curve instead for each option acros… view at source ↗
Figure 2
Figure 2. Ability information beyond binary correctness. (a) Mean item-level Fisher information in binary correctness and the additional information contributed by distractor identity. The full stacked length represents the information in the complete option response; percentages denote its increase over binary scoring. (b) Average number of detected informative distractors per item versus benchmark publication date. More rec… view at source ↗
Figure 3
Figure 3. Data-efficient LLM evaluation and benchmark calibration. (a) Kendall correlation between rankings from information-selected subsets and the full item bank. LLM-NRM performs best, especially at small item budgets. (b) Held￾out correctness log-loss after calibration with fewer LLMs. LLM-NRM produces useful item parameters with only 5 LLMs and approaches full-fleet performance near 40 MCQ items. Bands show variation ac… view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 45 canonical work pages

  1. [1]

    Darrell , title =

    Bock, R. Darrell , title =. Psychometrika , year =

  2. [2]

    Psychometrika , year =

    Thissen, David and Steinberg, Lynne , title =. Psychometrika , year =

    work page

  3. [3]

    , title =

    Suh, Youngsuk and Bolt, Daniel M. , title =. Psychometrika , year =

    work page

  4. [4]

    Measurement: Interdisciplinary Research and Perspectives , year =

    von Davier, Matthias , title =. Measurement: Interdisciplinary Research and Perspectives , year =

    work page

  5. [5]

    , title =

    Penfield, Randall D. , title =. Educational Measurement: Issues and Practice , year =

    work page

  6. [6]

    Findings of the Association for Computational Linguistics: EMNLP 2023 , year =

    Zhu, Chiwei and Xu, Benfeng and Wang, Quan and Zhang, Yongdong and Mao, Zhendong , title =. Findings of the Association for Computational Linguistics: EMNLP 2023 , year =

    work page 2023

  7. [7]

    Findings of the Association for Computational Linguistics: NAACL 2024 , pages=

    Large language models sensitivity to the order of options in multiple-choice questions , author=. Findings of the Association for Computational Linguistics: NAACL 2024 , pages=

    2024

  8. [8]

    Journal of Educational Measurement , volume=

    Guess where: The position of correct answers in multiple-choice test items as a psychometric variable , author=. Journal of Educational Measurement , volume=. 2003 , publisher=

    work page 2003

Show all 66 references
  1. [9]

    arXiv preprint arXiv:2407.06071 , year =

    Ivgi, Maor and Yoran, Ori and Berant, Jonathan and Geva, Mor , title =. arXiv preprint arXiv:2407.06071 , year =

    arXiv

  2. [10]

    and Linn, Robert L

    Strandmark, Nancy L. and Linn, Robert L. , title =. Applied Psychological Measurement , year =

  3. [11]

    Applied Psychological Measurement , volume=

    A comprehensive IRT approach for modeling binary, graded, and continuous responses with error in persons and items , author=. Applied Psychological Measurement , volume=. 2019 , publisher=

    2019

  4. [12]

    Marketing science , volume=

    The generalized multinomial logit model: accounting for scale and coefficient heterogeneity , author=. Marketing science , volume=. 2010 , publisher=

    2010

  5. [13]

    Transportation , volume=

    Can scale and coefficient heterogeneity be separated in random coefficients models? , author=. Transportation , volume=. 2012 , publisher=

    2012

  6. [14]

    , title =

    Guo, Chuan and Pleiss, Geoff and Sun, Yu and Weinberger, Kilian Q. , title =. Proceedings of the 34th International Conference on Machine Learning , year =

  7. [15]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics , year =

    Balepur, Nishant and Ravichander, Abhilasha and Rudinger, Rachel , title =. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics , year =

  8. [16]

    Statistical Theories of Mental Test Scores , editor =

    Birnbaum, Allan , title =. Statistical Theories of Mental Test Scores , editor =. 1968 , pages =

    1968

  9. [17]

    Measurement: Interdisciplinary Research and Perspectives , volume=

    On interpreting the model parameters for the three parameter logistic model , author=. Measurement: Interdisciplinary Research and Perspectives , volume=. 2009 , publisher=

    2009

  10. [18]

    On the Unidentifiability of the Fixed-Effects

    San Mart. On the Unidentifiability of the Fixed-Effects. Psychometrika , year =

  11. [19]

    and Lord, Frederic M

    Barton, Mark A. and Lord, Frederic M. , title =. 1981 , number =

    1981

  12. [20]

    Philip and Skene, Allan M

    Dawid, A. Philip and Skene, Allan M. , title =. Journal of the Royal Statistical Society: Series C (Applied Statistics) , year =

  13. [21]

    Advances in Neural Information Processing Systems 23 , year =

    Welinder, Peter and Branson, Steve and Belongie, Serge and Perona, Pietro , title =. Advances in Neural Information Processing Systems 23 , year =

  14. [22]

    Advances in Neural Information Processing Systems 37, Datasets and Benchmarks Track , year =

    Wang, Yubo and Ma, Xueguang and Zhang, Ge and Ni, Yuansheng and Chandra, Abhranil and Guo, Shiguang and Ren, Weiming and Arulraj, Aaran and He, Xuan and Jiang, Ziyan and others , title =. Advances in Neural Information Processing Systems 37, Datasets and Benchmarks Track , year =

  15. [23]

    , title =

    Rein, David and Hou, Betty Li and Stickland, Asa Cooper and Petty, Jackson and Pang, Richard Yuanzhe and Dirani, Julien and Michael, Julian and Bowman, Samuel R. , title =. First Conference on Language Modeling , year =

  16. [24]

    arXiv preprint arXiv:1803.05457 , year =

    Clark, Peter and Cowhey, Isaac and Etzioni, Oren and Khot, Tushar and Sabharwal, Ashish and Schoenick, Carissa and Tafjord, Oyvind , title =. arXiv preprint arXiv:1803.05457 , year =

    arXiv

  17. [25]

    Findings of the Association for Computational Linguistics: NAACL 2024 , year =

    Zhong, Wanjun and Cui, Ruixiang and Guo, Yiduo and Liang, Yaobo and Lu, Shuai and Wang, Yanlin and Saied, Amin and Chen, Weizhu and Duan, Nan , title =. Findings of the Association for Computational Linguistics: NAACL 2024 , year =

    2024

  18. [26]

    Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics , year =

    Talmor, Alon and Herzig, Jonathan and Lourie, Nicholas and Berant, Jonathan , title =. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics , year =

    2019

  19. [27]

    IEEE/ACM Transactions on Audio, Speech, and Language Processing , volume=

    Logiqa 2.0—an improved dataset for logical reasoning in natural language understanding , author=. IEEE/ACM Transactions on Audio, Speech, and Language Processing , volume=. 2023 , publisher=

    2023

  20. [28]

    Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , year =

    Mihaylov, Todor and Clark, Peter and Khot, Tushar and Sabharwal, Ashish , title =. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , year =

    2018

  21. [29]

    Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics , year =

    Lin, Stephanie and Hilton, Jacob and Evans, Owain , title =. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics , year =

  22. [30]

    Applied Sciences , year =

    Jin, Di and Pan, Eileen and Oufattole, Nassim and Weng, Wei-Hung and Fang, Hanyi and Szolovits, Peter , title =. Applied Sciences , year =

  23. [31]

    Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , year =

    Lai, Guokun and Xie, Qizhe and Liu, Hanxiao and Yang, Yiming and Hovy, Eduard , title =. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , year =

    2017

  24. [32]

    Social IQa: Commonsense reasoning about social interactions , author=. Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP) , pages=

    2019

  25. [33]

    Proceedings of the 34th AAAI Conference on Artificial Intelligence , year =

    Khot, Tushar and Clark, Peter and Guerquin, Michal and Jansen, Peter and Sabharwal, Ashish , title =. Proceedings of the 34th AAAI Conference on Artificial Intelligence , year =

  26. [34]

    International Conference on Learning Representations , year =

    Yu, Weihao and Jiang, Zihang and Dong, Yanfei and Feng, Jiashi , title =. International Conference on Learning Representations , year =

  27. [35]

    International Conference on Learning Representations , year =

    Hendrycks, Dan and Burns, Collin and Basart, Steven and Zou, Andy and Mazeika, Mantas and Song, Dawn and Steinhardt, Jacob , title =. International Conference on Learning Representations , year =

  28. [36]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    MetaEval: Measuring the Discrimination of Benchmarks for Efficient LLM Evaluation , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  29. [37]

    Probabilistic models for some intelligence and attainment tests

    Studies in mathematical psychology: I. Probabilistic models for some intelligence and attainment tests. , author=. 1960 , publisher=

    1960

  30. [38]

    1968 , publisher=

    Statistical Theories of Mental Test Scores , author=. 1968 , publisher=

    1968

  31. [39]

    Proceedings of the 12th International Conference on Educational Data Mining , year =

    Yeung, Chun-Kit , title =. Proceedings of the 12th International Conference on Educational Data Mining , year =

  32. [40]

    Chen, Yu and Silva Filho, Telmo and Prudencio, Ricardo B and Diethe, Tom and Flach, Peter , booktitle=. ^. 2019 , organization=

    2019

  33. [41]

    Annual Meeting of the Association for Computational Linguistics , year=

    Comparing test sets with item response theory , author=. Annual Meeting of the Association for Computational Linguistics , year=

  34. [42]

    2025 , eprint=

    Position: AI Evaluation Should Learn from How We Test Humans , author=. 2025 , eprint=

    2025

  35. [43]

    arXiv preprint arXiv:2606.15643 , year=

    Extending Item Response Theory for Efficient and Meaningful Multilingual Evaluation , author=. arXiv preprint arXiv:2606.15643 , year=

  36. [44]

    arXiv preprint arXiv:2511.04689 , year=

    Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks , author=. arXiv preprint arXiv:2511.04689 , year=

    arXiv

  37. [45]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Lost in benchmarks? rethinking large language model benchmarking with item response theory , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  38. [46]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    The llm already knows: Estimating llm-perceived question difficulty via hidden representations , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  39. [47]

    arXiv preprint arXiv:2207.05221 , year=

    Language models (mostly) know what they know , author=. arXiv preprint arXiv:2207.05221 , year=

    arXiv

  40. [48]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    2023

  41. [49]

    arXiv preprint arXiv:2509.10625 , year=

    No answer needed: Predicting llm answer accuracy from question-only linear probes , author=. arXiv preprint arXiv:2509.10625 , year=

  42. [50]

    Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    PredictaBoard: Benchmarking LLM score predictability , author=. Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    2025

  43. [51]

    arXiv preprint arXiv:2605.30504 , year=

    Auditing LLM Benchmarks with Item Response Theory , author=. arXiv preprint arXiv:2605.30504 , year=

    arXiv

  44. [52]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    Estimating llm consistency: A user baseline vs surrogate metrics , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  45. [53]

    arXiv preprint arXiv:2606.17312 , year=

    Quantifying consistency in LLM logical reasoning via structural uncertainty , author=. arXiv preprint arXiv:2606.17312 , year=

    arXiv

  46. [54]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Calibrating large language models with sample consistency , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  47. [55]

    International Conference on Learning Representations , volume=

    Large language models are not robust multiple choice selectors , author=. International Conference on Learning Representations , volume=

  48. [56]

    Option symbol matters: Investigating and mitigating multiple-choice option symbol bias of large language models , author=. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (V...

    2025

  49. [57]

    arXiv preprint arXiv:2403.04132 , year=

    Chatbot arena: An open platform for evaluating llms by human preference , author=. arXiv preprint arXiv:2403.04132 , year=

    arXiv

  50. [58]

    arXiv preprint arXiv:2606.07422 , year=

    The Masked Advantage: Uncovering Local-Language Access to Cultural Knowledge in LLMs , author=. arXiv preprint arXiv:2606.07422 , year=

    arXiv

  51. [59]

    Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pages =

    Building an Evaluation Scale using Item Response Theory , author =. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pages =. 2016 , publisher =. doi:10.18653/v1/D16-1062 , url =

    2016 doi

  52. [60]

    Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages =

    Efficient Benchmarking (of Language Models) , author =. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages =. 2024 , publisher =. doi:10.18653/v1/2024.na...

    2024 doi

  53. [61]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards , author =. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =. 2024 , publisher =. doi:10.18653/v1/2024.acl-long....

    2024 doi

  54. [62]

    Beyond Accuracy: Behavioral Testing of

    Ribeiro, Marco Tulio and Wu, Tongshuang and Guestrin, Carlos and Singh, Sameer , booktitle =. Beyond Accuracy: Behavioral Testing of. 2020 , publisher =. doi:10.18653/v1/2020.acl-main.442 , url =

    2020 doi

  55. [63]

    The Hitchhiker

    Dror, Rotem and Baumer, Gili and Shlomov, Segev and Reichart, Roi , booktitle =. The Hitchhiker. 2018 , publisher =. doi:10.18653/v1/P18-1128 , url =

    2018 doi

  56. [64]

    The Spanish Journal of Psychology , volume=

    Multiple-choice tests: Polytomous IRT models misestimate item information , author=. The Spanish Journal of Psychology , volume=. 2014 , publisher=

    2014

  57. [65]

    arXiv preprint arXiv:2402.14992 , year=

    tinyBenchmarks: evaluating LLMs with fewer examples , author=. arXiv preprint arXiv:2402.14992 , year=

    arXiv

  58. [66]

    International Conference on Learning Representations , volume=

    metabench-A sparse benchmark of reasoning and knowledge in large language models , author=. International Conference on Learning Representations , volume=

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.