Pith. sign in

REVIEW 3 major objections 5 minor 28 references

Option-ID Based Elimination For Multiple Choice Questions

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Pith's one-line read Option-ID elimination lifts LLM accuracy on multiple-choice questions.

desk verdict Useful inference-time trick for MCQ evaluation; the debiasing prior's provenance is underspecified but the central comparison to D-MCP is fair. read the letter →

arxiv 2501.15175 v3 pith:WD3BNXM4 submitted 2025-01-25 cs.CL cs.AIcs.LG

classification cs.CLcs.AIcs.LG
keywords multiplechoicequestionsprocessofeliminationoption-IDprobabilitytokenbiasdebiasingzero-shotevaluationfew-shotlearninglargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the classic test-taking strategy of process of elimination can be made to work for large language models if you apply it to the option labels (A, B, C) rather than to the option text. It proposes PoE_ID, which ranks option IDs by debiased log-probability and removes the weakest ones before making a final choice. The authors report consistent gains over nine baselines across six open-weight LLMs and four datasets, with the largest improvements on questions that have more options. The debiasing step is central: without it, the method's gains vanish when answer labels are shuffled. The result matters because multiple-choice evaluation is widespread, and a cheap inference-time recipe that improves accuracy could make benchmark scores more reliable.

What carries the argument

The load-bearing object is the debiasing ratio $P_d(o_i \mid q, x) = P(o_i \mid q, x) / P_{\text{prior}}(o_i)$, where $P_{\text{prior}}(o_i)$ is a prior over option IDs estimated from a small set of additional samples. This corrects for token bias, the tendency of LLMs to assign a priori higher probability to certain letters, before elimination thresholds are applied. The two strategies, PoE_log_ID (threshold at average log-probability) and PoE_seq_ID (iterative removal of the lowest-probability ID), both run on these debiased ID probabilities.

What would settle it

Run PoE_log_ID on a new MCQ benchmark with answer labels deliberately rebalanced so that one option ID is the correct answer much more often than others, using a prior estimated only from labeled training data rather than the test set. If accuracy drops to or below the debiased direct-ID baseline, the claimed out-of-the-box improvement is an artifact of transductive prior estimation.

Watch

Extended reading notes

Core claim

The central claim is that a process of elimination applied to option IDs, not option texts, is a reliable way to raise LLM accuracy on multiple-choice questions, provided the raw ID probabilities are first debiased against token bias. Concretely, PoE_log_ID computes the debiased log-probability of each option ID, eliminates every option whose ID falls below the average, re-scores the surviving IDs, and picks the highest. Across six LLMs and four datasets, the paper reports this consistently outperforms existing PoE variants, [MASK] replacement, and direct ID selection, and that the gains grow with the number of options.

Load-bearing premise

The method requires a small set of additional examples to estimate the option-ID prior, but the paper never reports where those examples come from, how many there are, or that they are disjoint from the test set; if that prior is estimated on the evaluation benchmark itself, the reported gains may not survive on a fresh set of questions with a different distribution of answer labels.

Editorial extensions

If this is right

  • PoE_log_ID improves zero-shot accuracy on datasets with many options (10-option ABU and MMLU-Pro) over the debiased direct ID selection baseline D-MCP.
  • The debiasing step is necessary; without it, naive ID-based elimination fails when answer labels are randomized, as shown in Table 1.
  • Option elimination outperforms [MASK] replacement because LLMs still allocate probability mass to masked IDs, distorting the distribution over surviving options.
  • LLMs are inherently poor at explicitly identifying incorrect options, so direct elimination (EE) underperforms the proposed ID-based approach.
  • The gains persist in few-shot settings with K=1 and K=10 demonstrations, supporting the method's robustness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the prior is estimated on the evaluation benchmark itself, the reported zero-shot gains are transductive and may not transfer to a fresh set of MCQs with a different distribution of answer labels; an independent prior-estimation study would settle this.
  • Because the method operates on letter IDs, changing the mapping from letters to options (e.g., shuffling choices) could alter the prior and thus the gains, so the method's effectiveness may be partly tied to conventional A/B/C/D ordering.
  • A testable extension is to estimate the option-ID prior on a held-out split of each dataset and compare accuracy against the paper's reported numbers; if the gap closes, the method's practical value outside the benchmark setting would be clearer.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes PoE_ID, a process-of-elimination method for multiple-choice questions that operates on option IDs rather than option text. Two elimination strategies are introduced: PoE_log_ID, which removes options whose debiased log-probability falls below the average, and PoE_seq_ID, which iteratively removes the lowest-probability option. The debiasing step adopts Pride (Zheng et al., 2023a) to correct token bias using a small set of additional samples. Experiments across 6 LLMs and 4 datasets (Timedial, Arithmetic, ABU, MMLU-Pro) compare against 9 baselines, reporting accuracy gains for both zero-shot and few-shot settings, with the largest gains on datasets with more options. Additional analyses examine confidence changes after elimination, comparison with [MASK] replacement, and reasons for failure of explicit incorrect-option identification.

Significance. If the reported results are robust, the paper offers a simple, inference-time, and inexpensive method to improve LLM accuracy on MCQs, especially when many options are present. The method is intuitive, the code is released, and the comparison against a wide set of baselines is a strength. The analyses in Section 5.2 and Section 5.3 provide useful mechanistic insight into why eliminating options can help. However, the central performance claim rests on an incompletely specified debiasing prior, and the absence of statistical testing leaves the magnitude and consistency of gains unclear. These issues must be resolved before the contribution can be fully assessed.

major comments (3)
  1. [§3.3, §4.1, §5.5] The debiasing prior P_prior(oi) is described only as estimated from 'a small set of additional samples,' with no size, source, or split reported. Section 5.5 states that few-shot demonstrations are drawn from 'additional samples used for computing P_prior(oi)', and Section 4.1 describes sampling 1000 instances 'prioritizing those from the test set' without identifying any other data source. This creates a serious risk that the prior is estimated on the evaluation benchmark itself, making the zero-shot results transductive (the method sees unlabeled test-distribution statistics) and the few-shot results potentially label-leaking. Because every claimed improvement over D-MCP depends on this prior, the paper must report: (i) the number and provenance of the prior samples, (ii) whether they are disjoint from the 1000 evaluation instances, and (iii) the exact formula used to compute P_prior. A concrete test would be to re-estimate the prior from a held-out set or from a different distribution and report whether the accuracy gains persist.
  2. [§5.1, Table 2] The headline claim of 'consistent state-of-the-art performance' is based on single point estimates with no significance tests, confidence intervals, or variance estimates. Many differences between PoE_log_ID and D-MCP are small in absolute terms (e.g., Timedial: 83.2 vs 83.1 for Gemma-3-12B; ABU: 73.8 vs 72.4 for Gemma-2-9B-It), and the 1000-instance sample could easily yield such gaps by chance. The paper should report paired bootstrap confidence intervals or at least standard errors across random subsamples, and ideally run multiple seeds for the few-shot and sampling-based conditions.
  3. [§5.1, Table 2 and Table 6] The claim that the method improves 'particularly in datasets with more options' conflates option count with dataset identity and domain difficulty. Timedial has 3 options but is a different task from Arithmetic (7 options) and ABU/MMLU-Pro (10 options). The average gains over D-MCP are not monotonic in option count (0.8, 6.3, 2.1, 2.6 points respectively), so the data do not directly support the stated relationship. To support the option-count claim, the authors should either vary the number of options within the same question set or provide a controlled comparison that isolates N from dataset content.
minor comments (5)
  1. [Abstract / Table 1] Table 1 in the Introduction uses 'MCP' before its definition in Section 4.3.2; consider defining it earlier or moving the table after the methodology section.
  2. [§5.3] The sentence 'The new strategy is based on the strategy mentioned in Section 3.4.1, which replace option with [MASK] instead of eliminating it' is confusing because Section 3.4.1 describes elimination, not mask replacement. Please rephrase and define MASK clearly.
  3. [§3.4.1 / §4.3.4] The notation yeli in Section 3.4.1 is not introduced; clarify that it is the option text corresponding to the lowest-ID-probability option. Also, the 'Incorrect option' definition in Section 5.4 uses 'arg max' for an incorrect option, which may be a typo for 'arg min'.
  4. [Various] There are several typos and formatting issues, e.g., 'avaliable' in the footnote, the inconsistent rendering of PoE_ID (sometimes PoEID, sometimes PoE_ID), and 'thePoEseqID' in Section 5.1. A careful proofread is needed.
  5. [§5.2, Figure 2] The figure caption does not report the error bars or the number of samples used for the 'proportion' and 'average increase' metrics. Please specify whether these are computed on all 1000 instances or a subset.

Circularity Check

1 steps flagged · score 2.0 of 10

Main accuracy claim is independent of external baselines; only the Section 5.2 confidence-increase analysis is self-referential because surviving options are renormalized.

  1. self definitional [Section 5.2, 'Analysis of the Effectiveness of PoE_log_ID'; recomputation step in Section 3.4.2]
    "For the second scenario, we investigate whether eliminating partial noise enhances the LLMs’ confidence in the correct option. To this end, we evaluate two metrics: (1) the proportion of samples where the LLMs’ probability for the correct option’s ID increases post-elimination compared to pre-elimination, and (2) the average increase in probability across all samples."

    The 'confidence increase' is measured on probabilities recomputed after the below-average options have been removed. Section 3.4.2 says 'we recompute the probabilities for the option IDs based on the xnew'; after removal, the surviving option-ID probabilities are renormalized over a smaller set. Thus, whenever the correct option survives elimination, its post-elimination probability is mechanically larger than its pre-elimination value in the full set, regardless of whether the model's true confidence changed. The >70% 'increase' is therefore a consequence of the elimination-and-renormalization definition, not independent evidence of noise reduction.

full rationale

The central claim—that PoE_ID improves zero-shot and few-shot MCQ accuracy—is evaluated against external baselines (MCP, D-MCP, IE, TG, etc.) and is not defined in terms of the method's own outputs. The debiasing prior P_prior(oi) is estimated from additional samples and used before computing option-ID probabilities; although Section 5.5 reveals that the same 'additional samples' also supply few-shot demonstrations, this is a data-provenance / potential-leakage concern, not a circularity, because the prior is not fitted to the reported accuracy metric. No load-bearing self-citation chain or imported uniqueness theorem appears. The only self-referential element is the Section 5.2 confidence-increase analysis: since the method removes low-probability options and recomputes/renormalizes the surviving ID probabilities, an apparent 'increase' in the correct option's probability is partly forced by construction. This analysis is supportive, not load-bearing for the main comparison, so the overall circularity score is low.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The method relies on data-derived priors and several unstated modeling assumptions, but introduces no new physical or mathematical entities. The central claim is empirical, so the main risks are the debiasing prior's provenance and the threshold heuristic.

free parameters (2)
  • Debiasing prior estimation sample count
    Section 3.3 says 'a small set of additional samples' without specifying the count; the debiasing strength depends on this choice.
  • Temperature = 0.1
    Section 4.2 fixes temperature at 0.1; it shapes the probability distributions used for elimination and final selection.
assumptions (5)
  • domain assumption Debiased ID probability Pd(oi|q,x) = P(oi|q,x)/P_prior(oi) measures true model confidence in option i.
    Adopted from Zheng et al. (2023a) in Section 3.3; the entire elimination ranking depends on this correction being valid.
  • domain assumption Option IDs are valid proxies for the options themselves, so eliminating an ID is equivalent to eliminating the option.
    Section 3.2: 'we use the probabilities of the option IDs as a substitute' for option probabilities.
  • ad hoc to paper The average log-probability threshold separates incorrect options from correct ones well enough to improve accuracy.
    Section 3.4.2: the threshold is the per-question average of log ID probabilities; no principled argument is given for why below-average implies incorrect.
  • domain assumption Recomputing ID probabilities on the reduced option set yields a more reliable final prediction than the full-set distribution.
    Section 3.4.2 and Section 5.2: the method assumes the recomputed distribution over surviving IDs is less noisy.
  • domain assumption The additional samples used to estimate P_prior(oi) come from the evaluation benchmark, and doing so does not compromise the zero-shot claim.
    Section 5.5: shots are drawn from 'additional samples used for computing P_prior(oi)', implying same-benchmark estimation; the transductive nature is not discussed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Option-ID Based Elimination For Multiple Choice Questions." pith.science (2026). https://pith.science/paper/WD3BNXM4

@misc{pith2026250115175,
  author       = {Pith},
  title        = {Pith review of: Option-ID Based Elimination For Multiple Choice Questions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WD3BNXM4}},
  note         = {Machine review of arXiv:2501.15175}
}
abstract

Multiple choice questions (MCQs) are a popular and important task for evaluating large language models (LLMs). Based on common strategies people use when answering MCQs, the process of elimination (PoE) has been proposed as an effective problem-solving method. Existing PoE methods typically either have LLMs directly identify incorrect options or score options and replace lower-scoring ones with [MASK]. However, both methods suffer from inapplicability or suboptimal performance. To address these issues, this paper proposes a novel option-ID based PoE ($\text{PoE}_{\text{ID}}$). $\text{PoE}_{\text{ID}}$ critically incorporates a debiasing technique to counteract LLMs token bias, enhancing robustness over naive ID-based elimination. It features two strategies: $\text{PoE}_{\text{ID}}^{\text{log}}$, which eliminates options whose IDs have log probabilities below the average threshold, and $\text{PoE}_{\text{ID}}^{\text{seq}}$, which iteratively removes the option with the lowest ID probability. We conduct extensive experiments with 6 different LLMs on 4 diverse datasets. The results demonstrate that $\text{PoE}_{\text{ID}}$, especially $\text{PoE}_{\text{ID}}^{\text{log}}$, significantly improves zero-shot and few-shot MCQs performance, particularly in datasets with more options. Our analyses demonstrate that $\text{PoE}_{\text{ID}}^{\text{log}}$ enhances the LLMs' confidence in selecting the correct option, and the option elimination strategy outperforms methods relying on [MASK] replacement. We further investigate the limitations of LLMs in directly identifying incorrect options, which stem from their inherent deficiencies.

Figures

Figures reproduced from arXiv: 2501.15175 by the authors.

Figure 1
Figure 1. An illustration of PoElog ID for a multiple choice question. The log probabilities of IDs B and D fall below the average threshold, leading to options B and D being eliminated from the option space. machine reading comprehension. Therefore, how to improve LLMs’ performance on MCQs tasks has become an important area of research. When people solve MCQs, they often use a special strategy named the process of eliminatio… view at source ↗
Figure 2
Figure 2. The blue bars represent the proportion of [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 11 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Nishant Balepur, Shramay Palta, and Rachel Rudinger. 2024. https://doi.org/10.48550/arXiv.2311.07532 It's not easy being wrong: Large language models struggle with process of elimination reasoning . arXiv preprint arXiv:2311.07532

  4. [4]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

  5. [5]

    Yu, Qiang Yang, and Xing Xie

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2024. https://doi.org/10.1145/3641289 A survey on evaluation of large language models . ACM Trans. Intell. Syst. Technol., 15(3)

  6. [6]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. https://lmsys.org/blog/2023-03-30-vicuna/ Vicuna: An open-source chatbot impressing gpt-4 with 90\

  7. [7]

    Gallegos, Ryan A

    Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrabim Tanjim, Sunghul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. 2024. https://doi.org/10.48550/arXiv.2309.00770 Bias and fairness in large language models: A survey . arXiv preprint arXiv:2309.00770

  8. [8]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. In International Conference on Learning Representations

Show all 28 references
  1. [9]

    Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021. Surface form competition: Why the highest probability answer isn’t always right. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7038--7051, Online...

  2. [10]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems

  3. [11]

    Chenkai Ma and Xinya Du. 2023. Poe: Process of elimination for multiple choice reasoning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4487--4496, Singapore. Association for Computational Linguistics

  4. [12]

    Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Noisy channel language model prompting for few-shot text classification. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5316--5330...

  5. [13]

    OpenAI. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

  6. [14]

    Joshua Robinson and David Wingate. 2023. Leveraging large language models for multiple choice question answering. In The Eleventh International Conference on Learning Representations

  7. [15]

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal MdShoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, and et al. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transac...

  8. [16]

    Gemma Team. 2024 a . https://doi.org/10.34740/KAGGLE/M/3301 Gemma

  9. [17]

    Gemma Team. 2025. https://goo.gle/Gemma3Report Gemma 3

  10. [18]

    Qwen Team. 2024 b . https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

  11. [19]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023 a . Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

  12. [20]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 b . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

  13. [21]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems

  14. [22]

    Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023b. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations

  15. [23]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. 2024. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574

  16. [24]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems

  17. [25]

    Haoyan Yang, Yixuan Wang, Xingyin Xu, Hanyuan Zhang, and Yirong Bian. 2024. https://doi.org/10.48550/arXiv.2405.16856 Can we trust llms? mitigate overconfidence bias in llms through knowledge transfer . arXiv preprint arXiv:2405.16856

  18. [26]

    Tony Z Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pages 12697--12706. PMLR

  19. [27]

    Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2023 a . https://arxiv.org/abs/2309.03882 Large language models are not robust multiple choice selectors . arXiv preprint arXiv:2309.03882

  20. [28]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023 b . Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.