{"id":"8498e45d-6b7f-492e-adc7-d0c5dac4a2e1","arxiv_id":"2501.15175","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Eliminating options by comparing debiased letter-ID probabilities improves LLM accuracy on multiple choice benchmarks, most on questions with many options.","lead":"This paper proposes a new way for large language models to answer multiple choice questions: instead of picking the best answer directly, the model eliminates options one or several at a time based on the probabilities of their letter IDs. The method, called PoE_ID, reports consistent accuracy gains over prior methods, especially on questions with many choices.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Debiasing prior provenance is unreported; Section 5.5 ties prior samples to few-shot demos, implying the zero-shot result may be transductive and few-shot may leak evaluation examples.","rationale":"The reader already flags the transductive prior; I agree. The concern is amplified by Section 5.5's explicit statement that few-shot shots come from the same additional samples used for the prior, which makes evaluation-set provenance much more likely. The paper has independent support: the method's internal logic is sound—D-MCP's argmax is never below the average threshold, so PoE_log does not eliminate already-correct answers—and the comparison against IE_new/MASK controls for the debiasing factor. Thus the main threat is not the mechanism but the prior's data provenance. A disclosure plus a disjoint-prior ablation would settle it; if the ablation preserves the gains, the claim stands. Therefore no verdict change beyond the reader's CONDITIONAL is warranted.","tokens_in":12557,"tokens_out":4845,"duration_ms":46863,"concrete_test":"In the released code/config, recover the exact 'additional samples' split. Then rerun the Table 2 zero-shot comparison with P_prior estimated from a disjoint external calibration set (e.g., other BIG-Bench/MMLU-Pro questions not among the 1000 eval instances, or a fixed prompt like 'Answer:' with no options) and compare each row. In addition, rerun Table 5 with few-shot demonstrations drawn from a separate training/validation split rather than the prior pool. If PoE_log's margin over D-MCP does not survive, the central claim is transductive or leaked.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—PoE_log significantly improves zero-shot/few-shot MCQ accuracy—depends on the debiasing prior P_prior(oi) in Section 3.3. That prior is described only as estimated from 'a small set of additional samples'; no size, source, or split is given. Section 5.5 then states that few-shot demonstrations are drawn from 'additional samples used for computing P_prior(oi)', so the prior pool is the same pool used for few-shot. Since Section 4.1 describes sampling 1000 instances 'prioritizing those from the test set' and no other data source is identified, the prior may be fit to the evaluation benchmark itself. If so, the zero-shot comparison is transductive: P_prior adapts to the option-ID statistics and prompt format of the exact test distribution, and the reported gains over D-MCP may reflect this calibration rather than a general elimination strategy. For the few-shot results, the situation is worse if the demonstrations are labeled test instances. This unexamined data assumption is the most load-bearing because it directly underwrites the zero-shot/few-shot generalization claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PoE_ID, a process-of-elimination method for multiple-choice questions that operates on option IDs rather than option text. Two elimination strategies are introduced: PoE_log_ID, which removes options whose debiased log-probability falls below the average, and PoE_seq_ID, which iteratively removes the lowest-probability option. The debiasing step adopts Pride (Zheng et al., 2023a) to correct token bias using a small set of additional samples. Experiments across 6 LLMs and 4 datasets (Timedial, Arithmetic, ABU, MMLU-Pro) compare against 9 baselines, reporting accuracy gains for both zero-shot and few-shot settings, with the largest gains on datasets with more options. Additional analyses examine confidence changes after elimination, comparison with [MASK] replacement, and reasons for failure of explicit incorrect-option identification.","tokens_in":12825,"tokens_out":2487,"duration_ms":22909,"significance":"If the reported results are robust, the paper offers a simple, inference-time, and inexpensive method to improve LLM accuracy on MCQs, especially when many options are present. The method is intuitive, the code is released, and the comparison against a wide set of baselines is a strength. The analyses in Section 5.2 and Section 5.3 provide useful mechanistic insight into why eliminating options can help. However, the central performance claim rests on an incompletely specified debiasing prior, and the absence of statistical testing leaves the magnitude and consistency of gains unclear. These issues must be resolved before the contribution can be fully assessed.","major_comments":[{"comment":"The debiasing prior P_prior(oi) is described only as estimated from 'a small set of additional samples,' with no size, source, or split reported. Section 5.5 states that few-shot demonstrations are drawn from 'additional samples used for computing P_prior(oi)', and Section 4.1 describes sampling 1000 instances 'prioritizing those from the test set' without identifying any other data source. This creates a serious risk that the prior is estimated on the evaluation benchmark itself, making the zero-shot results transductive (the method sees unlabeled test-distribution statistics) and the few-shot results potentially label-leaking. Because every claimed improvement over D-MCP depends on this prior, the paper must report: (i) the number and provenance of the prior samples, (ii) whether they are disjoint from the 1000 evaluation instances, and (iii) the exact formula used to compute P_prior. A concrete test would be to re-estimate the prior from a held-out set or from a different distribution and report whether the accuracy gains persist.","section":"§3.3, §4.1, §5.5"},{"comment":"The headline claim of 'consistent state-of-the-art performance' is based on single point estimates with no significance tests, confidence intervals, or variance estimates. Many differences between PoE_log_ID and D-MCP are small in absolute terms (e.g., Timedial: 83.2 vs 83.1 for Gemma-3-12B; ABU: 73.8 vs 72.4 for Gemma-2-9B-It), and the 1000-instance sample could easily yield such gaps by chance. The paper should report paired bootstrap confidence intervals or at least standard errors across random subsamples, and ideally run multiple seeds for the few-shot and sampling-based conditions.","section":"§5.1, Table 2"},{"comment":"The claim that the method improves 'particularly in datasets with more options' conflates option count with dataset identity and domain difficulty. Timedial has 3 options but is a different task from Arithmetic (7 options) and ABU/MMLU-Pro (10 options). The average gains over D-MCP are not monotonic in option count (0.8, 6.3, 2.1, 2.6 points respectively), so the data do not directly support the stated relationship. To support the option-count claim, the authors should either vary the number of options within the same question set or provide a controlled comparison that isolates N from dataset content.","section":"§5.1, Table 2 and Table 6"}],"minor_comments":[{"comment":"Table 1 in the Introduction uses 'MCP' before its definition in Section 4.3.2; consider defining it earlier or moving the table after the methodology section.","section":"Abstract / Table 1"},{"comment":"The sentence 'The new strategy is based on the strategy mentioned in Section 3.4.1, which replace option with [MASK] instead of eliminating it' is confusing because Section 3.4.1 describes elimination, not mask replacement. Please rephrase and define MASK clearly.","section":"§5.3"},{"comment":"The notation yeli in Section 3.4.1 is not introduced; clarify that it is the option text corresponding to the lowest-ID-probability option. Also, the 'Incorrect option' definition in Section 5.4 uses 'arg max' for an incorrect option, which may be a typo for 'arg min'.","section":"§3.4.1 / §4.3.4"},{"comment":"There are several typos and formatting issues, e.g., 'avaliable' in the footnote, the inconsistent rendering of PoE_ID (sometimes PoEID, sometimes PoE_ID), and 'thePoEseqID' in Section 5.1. A careful proofread is needed.","section":"Various"},{"comment":"The figure caption does not report the error bars or the number of samples used for the 'proportion' and 'average increase' metrics. Please specify whether these are computed on all 1000 instances or a subset.","section":"§5.2, Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The key stumbling block is the provenance of the debiasing prior. If the prior is estimated on the test set, the zero-shot comparison is transductive and the few-shot results may leak labels, which would invalidate the central generalization claim. The authors must disclose the prior sample construction and rerun at least one main experiment with a properly separated prior to demonstrate that the gains are not an artifact of this design. The absence of any uncertainty quantification is also concerning for a paper that claims 'significantly improves' performance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick read on arXiv:2501.15175. The paper proposes PoE_ID: debias option-ID probabilities (Pride) and then eliminate options either below the average log-probability or sequentially by lowest probability. The new bit is the specific combination—nobody has used debiased option-ID probabilities as the elimination signal. The components (MCP, Pride, IE) predate it, but the combination and the two elimination strategies are legitimate extensions, and the authors are honest that naive ID elimination fails on non-uniform labels.\n\nWhat it does well: the experiments are broad—6 LLMs, 4 datasets, 9 baselines. PoE_log beats D-MCP consistently across essentially every cell in Table 2, which is the fair comparison since both use the same debiasing prior. The analysis in Section 5.3 shows why elimination beats [MASK] replacement (probability mass still leaks to masked IDs), and Section 5.4 gives a plausible reason why explicit elimination fails. Code is available. That's reproducible work and should count.\n\nSoft spots, in order of importance. First, the debiasing prior P_prior(oi) is the crux of the method, but the paper never reports how many \"additional samples\" are used or where they come from. Section 5.5 says few-shot demonstrations are drawn from \"additional samples used for computing P_prior(oi)\", which raises the possibility that the prior is fit to the test distribution. If so, the zero-shot numbers are transductive and won't necessarily transfer to fresh MCQs with a different option-ID distribution. That said, this doesn't break the main comparison: D-MCP uses the same prior, so the PoE_ID vs D-MCP gain is not an artifact of the prior itself. It does undermine the \"consistent state-of-the-art\" framing versus the non-debiased baselines, and the few-shot setup could leak labels if the demonstrations are from the test set. The authors need to clarify split provenance. Second, no error bars or significance tests anywhere. The gains are a few points, and some cells (e.g., Timedial) are within noise. The word \"significantly\" is doing a lot of work. Third, the cross-dataset comparison attributes differences to option count, but dataset difficulty and option count are confounded—Timedial is easy, MMLU-Pro is hard. Minor.\n\nOn the stress-test note: it is right to flag the prior provenance as the key unanswered question, but I don't think it's a load-bearing flaw in the central claim. The claim of improvement over D-MCP survives even if the prior is transductive, because both methods share it. What you'd want in revision is transparency about the prior pool so the reader can judge whether the absolute numbers are fair.\n\nWho this is for: anyone building inference-time wrappers over open LLMs for MCQ evaluation, or using MCQ answers as a component in a larger pipeline. The gains are a few accuracy points, not a qualitative jump, and the method doesn't fix underlying reasoning limits.\n\nRecommendation: send it to peer review. The experiments are solid enough, the contribution is a real if modest extension, and the open questions are addressable in revision. A serious referee should ask for split details, significance testing, and a caveat on the SOTA claim.","headline":"Useful inference-time trick for MCQ evaluation; the debiasing prior's provenance is underspecified but the central comparison to D-MCP is fair.","tokens_in":13293,"tokens_out":2879,"would_cite":false,"duration_ms":25191,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Option-ID elimination lifts LLM accuracy on multiple-choice questions.","keywords":["multiple choice questions","process of elimination","option-ID probability","token bias debiasing","zero-shot evaluation","few-shot learning","large language models"],"falsifier":"Run PoE_log_ID on a new MCQ benchmark with answer labels deliberately rebalanced so that one option ID is the correct answer much more often than others, using a prior estimated only from labeled training data rather than the test set. If accuracy drops to or below the debiased direct-ID baseline, the claimed out-of-the-box improvement is an artifact of transductive prior estimation.","tokens_in":12394,"feed_emoji":"🎯","tokens_out":4887,"duration_ms":38194,"temperature":0.7,"pith_summary":"The paper argues that the classic test-taking strategy of process of elimination can be made to work for large language models if you apply it to the option labels (A, B, C) rather than to the option text. It proposes PoE_ID, which ranks option IDs by debiased log-probability and removes the weakest ones before making a final choice. The authors report consistent gains over nine baselines across six open-weight LLMs and four datasets, with the largest improvements on questions that have more options. The debiasing step is central: without it, the method's gains vanish when answer labels are shuffled. The result matters because multiple-choice evaluation is widespread, and a cheap inference-time recipe that improves accuracy could make benchmark scores more reliable.","feed_headline":"Option-ID elimination lifts LLM accuracy on multiple-choice questions","feed_subtitle":"Removing weak A/B/C/D candidates after debiasing improves zero- and few-shot results on four benchmarks.","key_machinery":"The load-bearing object is the debiasing ratio $P_d(o_i \\mid q, x) = P(o_i \\mid q, x) / P_{\\text{prior}}(o_i)$, where $P_{\\text{prior}}(o_i)$ is a prior over option IDs estimated from a small set of additional samples. This corrects for token bias, the tendency of LLMs to assign a priori higher probability to certain letters, before elimination thresholds are applied. The two strategies, PoE_log_ID (threshold at average log-probability) and PoE_seq_ID (iterative removal of the lowest-probability ID), both run on these debiased ID probabilities.","core_discovery":"The central claim is that a process of elimination applied to option IDs, not option texts, is a reliable way to raise LLM accuracy on multiple-choice questions, provided the raw ID probabilities are first debiased against token bias. Concretely, PoE_log_ID computes the debiased log-probability of each option ID, eliminates every option whose ID falls below the average, re-scores the surviving IDs, and picks the highest. Across six LLMs and four datasets, the paper reports this consistently outperforms existing PoE variants, [MASK] replacement, and direct ID selection, and that the gains grow with the number of options.","pith_inferences":["If the prior is estimated on the evaluation benchmark itself, the reported zero-shot gains are transductive and may not transfer to a fresh set of MCQs with a different distribution of answer labels; an independent prior-estimation study would settle this.","Because the method operates on letter IDs, changing the mapping from letters to options (e.g., shuffling choices) could alter the prior and thus the gains, so the method's effectiveness may be partly tied to conventional A/B/C/D ordering.","A testable extension is to estimate the option-ID prior on a held-out split of each dataset and compare accuracy against the paper's reported numbers; if the gap closes, the method's practical value outside the benchmark setting would be clearer."],"forward_implications":["PoE_log_ID improves zero-shot accuracy on datasets with many options (10-option ABU and MMLU-Pro) over the debiased direct ID selection baseline D-MCP.","The debiasing step is necessary; without it, naive ID-based elimination fails when answer labels are randomized, as shown in Table 1.","Option elimination outperforms [MASK] replacement because LLMs still allocate probability mass to masked IDs, distorting the distribution over surviving options.","LLMs are inherently poor at explicitly identifying incorrect options, so direct elimination (EE) underperforms the proposed ID-based approach.","The gains persist in few-shot settings with K=1 and K=10 demonstrations, supporting the method's robustness."],"supporting_citations":[{"why":"Supplies the Pride debiasing method that corrects token bias in option-ID probabilities, which the paper adopts as its core debiasing step.","marker":"(Zheng et al., 2023a)"},{"why":"Introduces the Multiple Choice Prompt (MCP), the direct option-ID selection baseline that PoE_ID extends and compares against.","marker":"(Robinson and Wingate, 2023)"},{"why":"Proposes the [MASK]-replacement PoE method (IE) that the paper uses as a baseline and analyzes in comparison to elimination.","marker":"(Ma and Du, 2023)"},{"why":"Documents that LLMs struggle with explicit elimination of incorrect options, motivating the option-ID-based elimination approach.","marker":"(Balepur et al., 2024)"},{"why":"Provides the MMLU-Pro dataset used for evaluation.","marker":"(Wang et al., 2024)"},{"why":"Provides the BIG-Bench subdatasets (Arithmetic, ABU, Timedial) used for evaluation.","marker":"(Srivastava et al., 2023)"}],"fun_headline_variants":["Debiased option-ID elimination boosts LLM MCQ accuracy","Eliminate low-ID-prob options to raise LLM MCQ scores","PoE_ID beats mask replacement for LLM multiple-choice","Remove weak option IDs to improve LLM test results","Option-ID process of elimination with debiasing wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method requires a small set of additional examples to estimate the option-ID prior, but the paper never reports where those examples come from, how many there are, or that they are disjoint from the test set; if that prior is estimated on the evaluation benchmark itself, the reported gains may not survive on a fresh set of questions with a different distribution of answer labels.","fun_headline_variants_meta":{"raw":{"variants":["Debiased option-ID elimination boosts LLM MCQ accuracy","Eliminate low-ID-prob options to raise LLM MCQ scores","PoE_ID beats mask replacement for LLM multiple-choice","Remove weak option IDs to improve LLM test results","Option-ID process of elimination with debiasing wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000199,"raw_usage":{"total_tokens":1391,"prompt_tokens":982,"completion_tokens":409,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":327}},"tokens_in":598,"tokens_out":409,"duration_ms":3976,"temperature":1.0,"reasoning_tokens":327,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:32:36.257944+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run PoE_log_ID on a new MCQ benchmark with answer labels deliberately rebalanced so that one option ID is the correct answer much more often than others, using a prior estimated only from labeled training data rather than the test set. If accuracy drops to or below the debiased direct-ID baseline, the claimed out-of-the-box improvement is an artifact of transductive prior estimation.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the BIG-Bench subdatasets (Arithmetic, ABU, Timedial) used for evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the Multiple Choice Prompt (MCP), the direct option-ID selection baseline that PoE_ID extends and compares against."},{"cited_title":"It's Not Easy Being Wrong: Large Language Models Struggle with Process of Elimination Reasoning","cited_arxiv_id":"2311.07532","evidence_quote":"Documents that LLMs struggle with explicit elimination of incorrect options, motivating the option-ID-based elimination approach."}],"review_version":1}