{"id":"fa844dca-2efe-4bde-b2de-a699cd1949f1","arxiv_id":"2608.02966","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"An option-level item response model for LLMs shows that the identity of wrong multiple-choice answers carries substantial information about model ability, improving ranking and enabling 770x benchmark compression.","lead":"Most LLM benchmarks score multiple-choice answers as right or wrong. This paper proposes a model that uses the full pattern of choices, including which wrong answers an LLM prefers, to estimate ability and to rank models with far fewer questions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Wrong-answer informativeness is validated only against the model's own full-information estimates; external Arena Elo is never applied to incorrect-only ability, so the central claim lacks independent support.","rationale":"The paper makes a genuine contribution: LLM-NRM is a well-specified extension of Bock's NRM with sharpness, positional, and fallback parameters, and the held-out prediction gains in Table 1 plus the full-response Arena Elo correlation in Table 2 are real evidence. However, the strongest claim about wrong answers specifically is not supported by external validation. Table 3's Incorrect-Only correlation with the full-information estimate uses fitted item parameters from the same model; this is a consistency check, not validation. The +101% Fisher Information figure is a property of any fitted nominal model once option discriminations are estimated, so it cannot by itself prove that natural-language distractors carry ability information. The reader's first-token proxy concern is legitimate, but even with perfect p_obs the central claim still needs an external anchor for incorrect-only estimates. A direct Arena-Elo correlation for Incorrect-Only theta would settle this. This is a missing empirical link, not a mathematical contradiction, so the CONDITIONAL verdict stands.","tokens_in":13478,"tokens_out":5367,"duration_ms":53528,"concrete_test":"Re-estimate held-out theta using only Incorrect-Only observations for the 48 LLMs matched to Arena.ai Elo, and compute Spearman correlation with the external Elo scores, exactly as Table 2 does for full responses. If the Incorrect-Only Spearman is substantially below the full-response 0.920 or is not significantly positive, then the wrong-answer ability signal is largely a model-internal artifact; if it remains high, the central claim receives independent support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim that incorrect-answer identity carries useful ability information rests mainly on two results: (i) the +101% Fisher Information gain in Figure 2(a), computed from the fitted LLM-NRM parameters, and (ii) the Incorrect-Only ability estimates recovering full-information estimates at Spearman 0.943 in Table 3. Both are internal to the model. The Fisher Information decomposition in Eq. (16) guarantees I_wrong >= 0 for any nominal response model with distinct option discriminations, so the +101% number is a mathematical consequence of the fitted NRM, not an empirical measurement about LLMs. Table 3 compares Incorrect-Only theta to the full-information theta obtained from the same fitted item parameters and the same likelihood; high correlation could reflect parameter/identifiability coupling rather than genuine distractor signal. The external Arena Elo validation in Table 2 is computed only for full responses, so it does not test the wrong-answer claim. Unless incorrect-only estimates are anchored to an external criterion, the central conclusion that wrong answers carry distinct and useful measurement information is underdetermined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces LLM-NRM, a nominal-response item response theory model that treats an LLM's full probability distribution over MCQ answer options as the psychometric response. The model augments Bock's NRM with per-LLM response sharpness, positional bias, and a difficulty-gated fallback process, and is fit by MAP estimation on soft cross-entropy to 189 LLMs and 31,554 items from 14 benchmarks. The central empirical claims are: (i) LLM-NRM predicts held-out LLM-item responses better than binary IRT and NRM baselines, especially in the option channel; (ii) its ability estimates correlate more strongly with the external Arena.ai Text Elo leaderboard (Spearman 0.920) than raw accuracy or baseline IRT abilities; (iii) distractor identity adds +101% Fisher information beyond binary correctness; (iv) ability estimated from incorrect responses alone recovers the full-information ability estimate with Spearman 0.943; and (v) 41 information-selected items preserve the full-bank ranking with Kendall's tau 0.85.","tokens_in":13725,"tokens_out":6469,"duration_ms":60606,"significance":"The paper addresses a real and timely problem: binary accuracy scoring discards option-level response structure in LLM MCQ evaluation. The scale of the study, the 5-fold cell-level cross-validation, the comparison against a wide set of IRT and deep-learning baselines, and the independent Arena Elo check for full-response ability are substantial strengths. If the option-level information claims survive closer scrutiny, the model would be a useful tool for benchmark analysis, ability estimation, and test compression. The main gap is that the specific claim that incorrect answers carry distinct and useful measurement information is not yet anchored to an external criterion: the held-out prediction and Elo validation are for the full response, while the incorrect-only evidence is internal to the fitted model. The paper is suitable for publication after these validation gaps are addressed.","major_comments":[{"comment":"The observed response p_obs_ji is defined as the first-token log-probability mass on each valid option, with options absent from top-20 API logits assigned zero mass. This measurement proxy is load-bearing: every fitted curve, Fisher-information computation, and compression result inherits any error in this construction. Options that share first tokens, for example 'the Sun' versus 'the Moon', are not separated by first-token probability mass, and top-20 truncation can drop probability mass that is relevant to the full option distribution. Please report the fraction of items with first-token collisions and provide a robustness check on a subset where full option probabilities are obtained by forced decoding, or state explicitly the conditions under which the first-token proxy is valid.","section":"Answer-choice distributions"},{"comment":"The incorrect-only ability estimates are validated only against the full-information theta from the same fitted model. The external Arena Elo validation in Table 2 uses full responses and therefore does not test the wrong-answer claim. Because the two estimates in Table 3 share item parameters and the same likelihood, their Spearman 0.943 could partly reflect parameter coupling rather than genuinely independent distractor signal. Please add an external anchor for incorrect-only ability, for example the Spearman correlation of incorrect-only theta with Arena Elo on the matched LLMs, or a held-out prediction experiment in which only incorrect responses are used to predict correctness on new items or new LLMs.","section":"Ability Estimation from Incorrect Responses, Table 3"},{"comment":"The decomposition I_NRM = I_binary + I_wrong with I_wrong >= 0 is a mathematical identity for any nominal-response model with option-specific discriminations; the +101% figure is therefore a property of the fitted parameters, not an independent empirical measurement about LLMs. Please present this quantity as a model-based result with uncertainty, for example by bootstrapping over fitted parameters, and pair it with a behavioral outcome such as the option-channel held-out gain in Table 1 to support the claim that wrong-answer identity is empirically useful.","section":"Fisher Information from Distractor Identity, Eq. (16), Figure 2(a)"},{"comment":"The text does not make explicit whether the item parameters and the information-based item selection exclude the target held-out LLM. If the item bank and the selection criterion are calibrated on the full fleet including the evaluated LLM, the reported 41-item Kendall tau of 0.85 is optimistic. Please specify the exact split: item calibration on training LLMs only, item selection on training LLMs only, and evaluation on held-out LLMs.","section":"Compression of MCQ Benchmarks, Figure 3"}],"minor_comments":[{"comment":"The abstract reports Spearman 0.943 while Table 3 reports 0.9428 +/- 0.0174; please align these numbers.","section":"Abstract and Table 3"},{"comment":"The distinction between the soft-channel and hard-channel NRM baselines is not defined before the table is presented; please add one sentence describing what each channel means.","section":"Table 1"},{"comment":"The correlation between informative-distractor count and benchmark publication date is based on only 14 points; the Pearson r = 0.87 should be interpreted cautiously and the analysis would benefit from a confidence interval.","section":"Figure 2(b)"},{"comment":"The descriptions of prior baselines PSN-IRT and SD-IR are brief; please include enough detail to make clear how they were adapted to the option-response setting.","section":"Experimental Setup"},{"comment":"The sentence stating that LLM-NRM estimates 2(K_i - 1) free parameters per MCQ item should clarify that this counts discriminations and intercepts after imposing the sum-to-zero identification constraints.","section":"Fitting"},{"comment":"The limitation paragraph mentions partial probability observations and multidimensional abilities as future work, but does not discuss the sensitivity of results to the choice of the N(0,1) ability prior; a brief comment on prior sensitivity would strengthen the paper.","section":"Discussion and Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a psychometric or LLM-evaluation venue. The main reason I am not recommending acceptance is the missing external validation for the incorrect-only ability claim and the unresolved measurement-proxy question for first-token probabilities; both are fixable within the manuscript's scope. I did not find evidence of a fatal flaw in the central derivation, and I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I'll cut to it. The real contribution here is LLM-NRM: Bock's Nominal Response Model extended with per-LLM response sharpness, positional bias, and a difficulty-gated fallback term, fitted to soft option distributions. That is new and it is a natural fit for how LLMs actually behave. The empirical base is large—189 LLMs, 31,554 items—and the held-out prediction gains are genuine: option-channel accuracy 0.707 vs 0.643 for the best binary baseline, log-loss 0.818 vs 0.988. The Arena Elo correlation of 0.920 for full responses is a legitimate external check and beats the alternatives. I have no quarrel with the modeling or the fit. This paper deserves a serious referee.\n\nWhere I agree with the stress-test: the headline claim that wrong-answer identity itself carries ability information is not yet independently supported. The +101% Fisher information gain is a mathematical consequence of fitting a nominal model—Eq. (16) guarantees a nonnegative wrong-answer term once option discriminations differ—so it is a property of the fitted parameters, not an empirical measurement about LLMs. The incorrect-only theta recovery in Table 3 is compared against the model's own full-information estimate, so the high correlation could reflect parameter coupling rather than genuine distractor signal. The external Arena Elo check is only run on full responses, so it does not cover the wrong-answer claim. To make the central conclusion stick, the incorrect-only estimates need to be validated against an external criterion—Arena Elo or something else—directly. That is a missing experiment, not a fatal flaw.\n\nThe other weakness is the observation channel: first-token log-probability mass on option tokens, with zero mass assigned to options absent from API top-20 logits. If options share first tokens or top-20 truncation drops real mass, every fitted curve inherits the distortion. The paper reports only 0.56% letter-mass leakage, which mitigates but does not eliminate the concern. Also, no code or data is released, and the difficulty-index linearization in Eq. (10) has no error analysis—both are minor at this stage but should be addressed. The citation pattern looks fine: Bock, 3PL identifiability, LLM positional bias—all properly grounded. The self-citations are to prior IRT-for-LLM work and are not inflated.\n\nWho this is for: anyone working on LLM evaluation or psychometric models of language models. The contribution is incremental but solid. My bottom line: send it to review. The authors should do the extra external validation of incorrect-only estimates and ideally release code, but the core modeling and held-out results deserve referee time.","headline":"Solid option-level IRT extension for LLM evaluation with real held-out gains, but the claim that wrong answers carry independent ability signal still needs an external anchor.","tokens_in":837,"tokens_out":1143,"would_cite":true,"duration_ms":24695,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An option-aware psychometric model that scores which wrong answer an LLM picks predicts unseen responses better and agrees more closely with human preference rankings, according to this paper.","keywords":["LLM evaluation","item response theory","nominal response model","multiple-choice benchmarks","distractor analysis","psychometrics","ability estimation","benchmark compression"],"falsifier":"Take a set of MCQ items in which the correct option and at least one distractor share the same first token, recompute observed option probabilities from the full token sequence rather than first-token mass, and compare LLM-NRM's held-out log-loss against the first-token-based estimates; if the gap shrinks or reverses, the first-token proxy is the load-bearing simplification.","tokens_in":13218,"feed_emoji":"📊","tokens_out":5547,"duration_ms":47790,"temperature":0.7,"pith_summary":"Multiple-choice benchmarks for LLMs score a response as right or wrong, throwing away which wrong option was chosen. This paper argues that the wrong answers are not equivalent errors, and that the full probability distribution an LLM places over the options is itself a measurement signal. It introduces LLM-NRM, an option-aware psychometric model of latent ability, item characteristics, and LLM-specific response behaviors. Across 189 LLMs and 31,554 items, the model predicts held-out responses better than binary item response models, and its ability estimates line up more closely with an external human-preference leaderboard. If correct, the result turns distractor choices from noise into data: benchmarking could be compressed hundreds of times and rank models without ever observing a correct answer.","feed_headline":"Scoring which wrong option LLMs pick beats binary right/wrong scoring","feed_subtitle":"Modeling the full distribution over answer choices improves held-out prediction and human-preference alignment.","key_machinery":"The central object is LLM-NRM, an extension of Bock's nominal response model in which each answer option has its own discrimination and intercept parameters, and the respondent model adds a per-LLM inverse-temperature sharpness, a positional-bias vector, and a difficulty-gated guessing fallback. The fallback gate is a logistic function of the gap between LLM ability and a differentiable item-difficulty index derived from the NRM option parameters, and it interpolates between ability-driven utilities and a per-LLM fallback distribution over positions. This joint model separates calibration sharpness, content-independent position preference, and difficulty-dependent guessing from ability and item content, which the paper argues prevents these LLM-specific behaviors from contaminating ability estimates.","core_discovery":"The paper establishes that the full categorical distribution an LLM assigns to multiple-choice options, not just the selected answer, behaves as a psychometric response. Modeling this distribution with an adapted nominal response model yields better held-out response prediction than binary IRT and conventional nominal baselines, and the resulting ability estimates agree more closely with external human-preference Elo rankings than raw accuracy or any of the competing latent-ability estimates. Distractor identity alone is shown to contribute about twice as much Fisher information per item as correctness, and incorrect responses alone recover full-information ability estimates with Spearman correlation around 0.943.","pith_inferences":["The Fisher information gain should persist if option probabilities are recomputed from full-sequence scores rather than first-token mass; testing this would reveal how much of the reported advantage rests on the first-token proxy.","The strong agreement with external human-preference Elo suggests option-level modeling could serve as a cheap proxy for pairwise human evaluation, which could be validated by predicting individual pairwise judgments rather than aggregate scores.","The finding that newer benchmarks contain more informative distractors hints that distractor construction is a rising design axis, potentially guiding automatic distractor generation by maximizing Fisher information.","The single latent dimension is a deliberate simplification; a multidimensional extension could separate factual knowledge from reasoning and might also detect prompt-order contamination through the estimated positional parameters."],"forward_implications":["Benchmark designers can use option-level item parameters to identify which distractors actually discriminate ability and to write more informative wrong answers.","A new LLM can be ranked reliably from as few as 41 selected items, a 770-fold reduction in evaluation cost.","Because error patterns alone recover ability at Spearman 0.943, settings where correct answers are hidden or partially observed still permit meaningful model measurement.","Separating positional bias and calibration sharpness from ability lets reported accuracy gaps be decomposed into knowledge differences versus response-style differences."],"supporting_citations":[{"why":"Supplies the nominal response model that LLM-NRM extends to LLM responses.","marker":"Bock 1972"},{"why":"Provides the external crowdsourced human-preference Elo scores used to validate the ability scale.","marker":"Chiang et al. 2024"},{"why":"Documents fallback behaviors of LLMs under uncertainty, motivating the difficulty-gated guessing mechanism.","marker":"Ivgi et al. 2024"},{"why":"Shows LLMs' sensitivity to option order, motivating the positional-bias parameter.","marker":"Zheng et al. 2024"},{"why":"Establishes that models differ in calibration sharpness, motivating the per-LLM inverse-temperature parameter.","marker":"Kadavath et al. 2022"},{"why":"Supplies the MMLU-Pro item bank and the observation that accuracy loses discriminative power near saturation.","marker":"Wang et al. 2024"},{"why":"Prior benchmark-compression baseline requiring around 100 anchor items, which LLM-NRM outperforms at 41 items.","marker":"Maia Polo et al. 2024"}],"fun_headline_variants":["Distractor choices carry double the test information in LLM benchmarks","Incorrect responses alone recover full LLM ability scores","LLM benchmarks: every wrong answer counts, not just the right one","Modeling wrong answers improves LLM ability estimates and predictions","Option-level psychometrics: wrong answers reveal more for LLM eval"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire model assumes that the observed option distribution, first-token log-probability mass on each answer option with zero for options missing from returned top-20 logits, faithfully represents the model's true preference among complete answers.","fun_headline_variants_meta":{"raw":{"variants":["Distractor choices carry double the test information in LLM benchmarks","Incorrect responses alone recover full LLM ability scores","LLM benchmarks: every wrong answer counts, not just the right one","Modeling wrong answers improves LLM ability estimates and predictions","Option-level psychometrics: wrong answers reveal more for LLM eval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000737,"raw_usage":{"total_tokens":3276,"prompt_tokens":913,"completion_tokens":2363,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":2291}},"tokens_in":529,"tokens_out":2363,"duration_ms":15414,"temperature":1.0,"reasoning_tokens":2291,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:53:53.776154+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of MCQ items in which the correct option and at least one distractor share the same first token, recompute observed option probabilities from the full token sequence rather than first-token mass, and compare LLM-NRM's held-out log-loss against the first-token-based estimates; if the gap shrinks or reverses, the first-token proxy is the load-bearing simplification.","supporting_citations":[{"cited_title":"Findings of the Association for Computational Linguistics: NAACL 2024 , year =","cited_arxiv_id":null,"evidence_quote":"Supplies the MMLU-Pro item bank and the observation that accuracy loses discriminative power near saturation."}],"review_version":2}