{"id":"c6d61254-905a-4c0c-94be-bf55cb58f01a","arxiv_id":"2508.10022","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A p-value reformulation of split conformal prediction for LLM multiple-choice QA achieves nominal miscoverage control on MMLU and MMLU-Pro.","lead":"This paper applies split conformal prediction to multiple-choice question answering by LLMs, using the frequency of each option across repeated sampled responses as a conformity score and framing the inclusion rule as a p-value. It reports that prediction sets cover the correct answer at user-set rates and shrink as the allowed error rate increases.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Discrete frequency scores make the strict-inequality p-value in Eq. (5) anti-conservative under ties; the claimed coverage guarantee Eq. (2) is not established.","rationale":"I agree with the reader's rejection, but my strongest concern is different from the exchangeability worry. Even if exchangeability holds exactly, the paper's Eq. (5) p-value is not valid when the nonconformity scores are discrete, which they are by design because they are empirical frequencies over 20 samples. The strict '>' comparison makes p-values too small for true labels that tie calibration scores, so Eq. (7)'s bound can fail and the marginal coverage guarantee in Eq. (2) is not proven. The same discrete/threshold issue causes the claimed algebraic equivalence between Eq. (3) and Eq. (4) to fail, as shown by the n=10, alpha=0.5 counterexample. The reader's point about untested exchangeability is legitimate but secondary: the paper's proof is internally inconsistent even in the ideal exchangeable setting. This reinforces the rejection rather than moving the verdict. A randomized tie-breaking scheme may repair the method, but the submitted manuscript does not provide it, so the central claim of provable risk control is not established.","tokens_in":7878,"tokens_out":15521,"duration_ms":163405,"concrete_test":"Simulate exchangeable scores on the discrete grid {0,0.05,...,1}: draw n=100 calibration scores and one true-label test score i.i.d. from a distribution with substantial probability mass at a single value (including the all-equal case, e.g., all scores 0.5). For alpha=0.1, run 10^5 trials and compare the empirical miscoverage of the Eq. (5) rule with that of the Eq. (1) split-conformal rule. If Eq. (5) miscoverage exceeds 0.1, the guarantee fails. Then replace the p-value by the randomized version with U~Uniform(0,1) in the equal-score count and confirm that coverage is restored to at least 0.9.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the p-value rule of Eqs. (5)-(7) provides the marginal coverage guarantee Eq. (2), relying on the p-value being super-uniform under exchangeability. However, the nonconformity scores are discrete by construction: S(x,y)=1-f_hat(y|x), where f_hat is an empirical frequency over 20 single-token generations, so S takes values only in {0,0.05,...,1}. Ties are therefore unavoidable. Eq. (5) defines p(y)=(sum_i 1{s_i>S(xtest,y)}+1)/(n+1), using a strict inequality that ignores tied calibration scores. Under exchangeability with a discrete score distribution, a test score that ties calibration scores does not yield the uniform rank needed for Eq. (7). Degenerate example: if all n calibration scores and the test score are equal (e.g., all 0.5), then p=1/(n+1) for the true label with probability 1. For alpha>=1/(n+1), P(p<=alpha)=1, so Eq. (7)'s bound is violated and the true label is always excluded, even though the split-conformal set of Eq. (1) would include it. The issue also appears in the claimed equivalence of Eqs. (3) and (4): with n=10, alpha=0.5, a test score equal to the 6th smallest calibration score satisfies Eq. (3) (count of s_i<=S is 6<=6) but gives p=5/11<0.5 in Eq. (4), so Eq. (4) excludes it. The paper needs randomized tie-breaking, e.g., p=(sum 1{s_i>S}+U*sum 1{s_i=S}+1)/(n+1) with U~Uniform(0,1), or a conservative '>=' convention, plus a proof of coverage under that rule. Neither is provided.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a conformal prediction (CP) framework for multiple-choice QA in which an LLM is treated as a black-box scorer via 20 single-token generations per question. Nonconformity scores S(x,y)=1-f_hat(y|x) are computed from empirical choice frequencies, calibration is performed per subject, and prediction sets are constructed by comparing a p-value (Eq. 5) with alpha. The paper claims that this p-value rule is equivalent to the split-CP quantile rule and therefore inherits the marginal coverage guarantee P(y*_test in C_alpha(x_test)) >= 1-alpha (Eq. 2). Experiments on MMLU and MMLU-Pro with several open models report empirical error rates close to or below alpha and a monotone decrease of average prediction set size with alpha.","tokens_in":8355,"tokens_out":9157,"duration_ms":89695,"significance":"If the coverage argument were correct, the p-value formulation would provide a useful bridge between hypothesis testing and CP for LLM uncertainty. The empirical study is broad and provides clear evidence that a standard split-CP baseline controls miscoverage on these benchmarks, and the APSS analysis is a useful descriptive tool. The manuscript does not ship code or machine-checked proofs, and the central derivation is not currently sound: the proposed p-value is anti-conservative under ties and the proof of Eq. (7) is missing. The paper's contribution as a 'significance testing-enhanced' framework is also weakened by the fact that the p-value rule is essentially a restatement of the split-CP inclusion rule.","major_comments":[{"comment":"Eqs. (3) and (4) are not algebraically equivalent. For n=10, alpha=0.5, the conformal threshold in Eq. (3) is the 6th smallest calibration score (k=ceil(11*0.5)=6), so a test score equal to that value satisfies Eq. (3) because #{s_i <= S_test}=6 <= 6. However, with the strict '>' in Eq. (5), p=(10-6+1)/11=5/11 <= 0.5, so Eq. (4) excludes the same label. The off-by-one arises because #{s_i > S_test} = n - #{s_i <= S_test}; the '+1' in Eq. (5) does not compensate for the strict inequality. Because the scores are discrete (multiples of 0.05 with P=20), ties are unavoidable; if all n calibration scores and s_test are equal, Eq. (5) gives p=1/(n+1), so for alpha >= 1/(n+1) the true label is excluded with probability 1, violating Eq. (2). The paper needs a conservative rule such as p=(#{s_i >= S_test}+1)/(n+1) or randomized tie-breaking, and a coverage proof for that rule.","section":"Sec. 3.3, Eqs. (3)-(4)"},{"comment":"Eq. (7) is not a proof of coverage. It uses an undefined N and merely restates the inequality that needs to be established. A rigorous argument must show that, under exchangeability of (s_1,...,s_n,s_{n+1}), the proposed p-value is super-uniform, i.e., P(p <= alpha) <= alpha for all alpha. In particular, the paper does not show that the strict '>' p-value of Eq. (5) is super-uniform; the equal-score example above shows it is not. This is a load-bearing gap because Eq. (7) is the link to the marginal guarantee Eq. (2).","section":"Sec. 3.3, Eq. (7)"},{"comment":"Algorithm 1 contains multiple undefined symbols and a malformed condition. Line 3 iterates over 'D' without defining it; line 4 uses 'y*' instead of y*_i; line 6 sorts '{s_i}_{i=1}^N' although N is never defined (the calibration size is n); line 9 has 'if P(...) <= alpha then continue', where the outer 'P' appears to be a typo and the action of 'continue' conflicts with the surrounding else branch. Eq. (3) also writes S(x_i,y) where the prose (and Eq. (4)) use S(x_test,y). These errors prevent the reader from determining the exact implemented procedure.","section":"Algorithm 1 and Eq. (3)"},{"comment":"Section 4.2 states that the empirical validation constructs prediction sets via Eq. (3), i.e., the standard split-CP quantile rule, rather than via the p-value decision rule of Eqs. (4)-(5). Consequently the experiments do not validate the paper's claimed significance-testing formulation. If the implementation actually used the p-value rule, the manuscript must say so explicitly and address the tie issue; as written, the empirical findings are consistent with a standard split-CP baseline and do not support the specific ST-CP claims.","section":"Sec. 4.2 vs. Sec. 3.3"}],"minor_comments":[{"comment":"'Typel error rate' should be 'Type I error rate'.","section":"Sec. 3.1"},{"comment":"The sentence 'all results are not significantly lower than our set significance level, but are lower than the set value within a small range' is unclear; no statistical significance test is reported to support 'not significantly lower'.","section":"Sec. 4.2"},{"comment":"Figure numbering is inconsistent: the text refers to Figures 2-4, but Figure 1 (box plots) appears before Figure 2; the captions and callouts should be normalized.","section":"Figures"},{"comment":"The choice P=20, temperature=1.0, top-p=0.9 is not justified and no ablation is provided. Since the scores are discrete, this choice directly affects the tie problem discussed in the major comments.","section":"Sec. 4.1.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is best viewed as an application of split conformal prediction with a p-value reinterpretation; the coverage error is local and fixable. However, the experiments validate Eq. (3) rather than the proposed p-value rule, so the revised version must either demonstrate the p-value formulation on its own or clearly frame the contribution as a pedagogical restatement. If the journal values new methodology, the author should also identify an operational advantage of the p-value view, such as handling of ties or multiple-testing corrections."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is split conformal prediction re-described as a significance-testing framework, and the central claim that the two are equivalent is mathematically wrong. The paper is not ready for publication as is.\n\nWhat's good: the experimental setup is honest and straightforward. The authors sample 20 tokens per question, compute empirical option frequencies, build calibration scores, and report error rates that track alpha across MMLU and MMLU-Pro for several off-the-shelf models. The APSS curves decreasing with alpha are exactly what conformal theory predicts. As an empirical exercise, it confirms known behavior of split CP, but nothing more.\n\nThe soft spots are serious. Equations (3) and (4) are not equivalent. The inclusion rule in Eq (3) counts calibration scores <= the test score, while the p-value in Eq (5) uses a strict inequality, dropping tied calibration scores from the count. For n=10, alpha=0.5, a test score tied with the 6th smallest calibration score satisfies Eq (3) but gives p=5/11<0.5, excluding it. Because the nonconformity scores are discrete (empirical frequencies over 20 draws), ties are generic. The p-value is a strict-smaller version of the standard conformal p-value, making rejection more likely and breaking coverage. Randomized tie-breaking or a >= convention would fix it, but neither is present. Equation (7) uses an undefined N and is not a proof; it simply states the target bound.\n\nThe novelty claim is also overstated. The conformal p-value in Eq (5) is standard construction, and the 'integration' with significance testing is a restatement. Existing work applying CP to LLM MCQA is not cited. Exchangeability is assumed without testing, which is normal for CP, but the additional assumption of iid single-token generations across 20 samples is also unexamined.\n\nOverall, the empirical results are consistent with theory only if the theory is corrected. As written, the load-bearing derivation fails, so the paper does not establish provable risk control. A desk rejection is appropriate. The core idea (frequency-based conformity scores for MCQA) is reasonable and could be salvaged with correct proofs and proper tie handling; the author should also release code.","headline":"Routine split-conformal application whose central equivalence is algebraically wrong; experiments are consistent but do not rescue the theory.","tokens_in":8802,"tokens_out":3681,"would_cite":false,"duration_ms":36783,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A p-value reformulation of conformal prediction gives black-box LLMs answer sets whose chance of missing the true option is bounded by a user-set α, under exchangeability.","keywords":["conformal prediction","significance testing","p-values","multiple-choice question answering","large language models","hallucination detection","uncertainty quantification","average prediction set size"],"falsifier":"Run a calibration-test split where calibration is confined to one set of subjects (e.g., STEM) and evaluation to another (e.g., humanities) with the same p-value rule; if the empirical miscoverage rate exceeds α beyond bootstrap sampling error, the exchangeability premise is violated and the claimed guarantee does not hold. A second check: fix calibration data and change only the decoding temperature or prompt template at test time, then observe whether coverage stays above $1-\\alpha$.","tokens_in":7797,"feed_emoji":"🎯","tokens_out":8194,"duration_ms":83921,"temperature":0.7,"pith_summary":"The paper claims that a significance-testing version of conformal prediction can control, at any user-specified level α, the probability that the correct answer is left out of a prediction set in multiple-choice question answering. The method treats repeated LLM outputs as votes, computes the empirical frequency of each option, and converts the standard conformal score into a p-value; including or excluding each option is then a hypothesis-testing decision. If correct, this gives LLM users a model-agnostic, distribution-free way to request bounded error rates, which matters for hallucination-prone high-stakes question answering. Experiments on MMLU and MMLU-Pro across several off-the-shelf LLMs show empirical miscoverage below α and monotone shrinkage of prediction sets as α rises.","feed_headline":"Conformal p-values cap wrong-answer risk in LLM quizzes","feed_subtitle":"By testing each answer option, black-box LLM outputs become answer sets with a user-set error ceiling.","key_machinery":"The load-bearing object is the conformal p-value. With calibration nonconformity scores $s_1,\\dots,s_n$ and a test score $S(x_{\\text{test}}, y)$ for a candidate option, $p(y) = \\frac{\\sum_{i=1}^n \\mathbf{1}\\{s_i > S(x_{\\text{test}}, y)\\} + 1}{n+1}$; the rule “include $y$ when $p(y) > \\alpha$ and exclude $y$ when $p(y) \\le \\alpha$” is algebraically equivalent to the conformal quantile construction. The nonconformity score is $S(x,y) = 1 - \\hat{f}(y|x)$, where $\\hat{f}$ is the observed frequency of option $y$ across repeated single-token generations, so only black-box LLM outputs are needed.","core_discovery":"The central claim is that the prediction set built by comparing a p-value to α is the same prediction set built by the conformal quantile rule, so it inherits the conformal guarantee $P(y^*_{\\text{test}} \\in C_\\alpha(x_{\\text{test}})) \\ge 1-\\alpha$ under exchangeability. For each candidate option $y$, the paper defines $p(y) = \\frac{\\sum_{i=1}^n \\mathbf{1}\\{s_i > S(x_{\\text{test}}, y)\\} + 1}{n+1}$, using nonconformity scores $s_i$ from calibration; $y$ is excluded when $p(y) \\le \\alpha$. On the true label, this p-value is the rank of the test score among calibration scores, which is the usual conformal p-value. The paper therefore claims that significance-testing-based conformal prediction i","pith_inferences":["The guarantee is marginal over the calibration-test draw; it does not promise per-subject or per-question coverage, so subject-level error boxes in the paper should be read as descriptive diagnostics, not guaranteed bounds.","The method is likely sensitive to the exchangeability assumption: calibrating on one subject set and testing on a different subject set, or changing decoding temperature or prompt template between calibration and deployment, could push empirical miscoverage above α. This is a direct test the paper does not run.","Because the derivation never uses the specific form of the nonconformity score beyond exchangeability, the same p-value recipe could be applied to other scores, such as semantic similarity or token-level likelihoods, to extend calibrated abstention from MCQA to open-ended generation.","The 20-sample temperature-1.0 decoding is treated as i.i.d.; if the model's sampling is not actually exchangeable across questions (context dependence, batch artifacts), the nominal α may understate true risk."],"forward_implications":["A practitioner can set α in advance and receive a prediction set whose expected chance of missing the correct option is at most α, with no access to model internals.","Risk and set size are coupled: increasing α monotonically shrinks average prediction set size, so APSS doubles as an interpretable uncertainty diagnostic.","The same split-conformal calibration procedure works across model families and dataset difficulties; the paper reports below-target miscoverage on both MMLU and MMLU-Pro at a 1:1 calibration-test split.","Because the p-value rule is exactly the conformal rule, significance-testing language (null hypothesis, Type I error) and conformal language (coverage, miscoverage) describe the same single knob."],"supporting_citations":[{"why":"Supplies the standard conformal prediction quantile construction whose equivalence with the p-value rule is the paper's core derivation.","marker":"[1]"},{"why":"Defines the hypothesized synergy between split conformal prediction and significance testing that the proposed method claims to realize.","marker":"[16]"},{"why":"Points to the MCQA task setting in which the framework is instantiated and evaluated.","marker":"[17]"},{"why":"Prior applications of split conformal prediction to filter LLM outputs; the paper positions its method as a continuation of this line.","marker":"[13, 10]"},{"why":"Prior significance-testing approaches to hallucination detection, including p-value and FDR-based testing, which the proposed method adapts.","marker":"[7, 11]"}],"fun_headline_variants":["P-values give LLM answer sets with error ceiling","Statistical test tames LLM guessing in multiple choice","LLM answer sets with provable risk via p-values","P-value method matches conformal guarantees for LLMs","Risk-controlled answer sets from LLM responses"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The whole coverage guarantee rests on the nonconformity scores of calibration and test examples being exchangeable; the paper assumes this by splitting within each subject and treating its 20 sampled responses as i.i.d., but does not test it, and if test questions come from a different subject or distribution the claimed bound $P(y^*_{\\text{test}} \\in C_\\alpha(x_{\\text{test}})) \\ge 1-\\alpha$ can fail.","fun_headline_variants_meta":{"raw":{"variants":["P-values give LLM answer sets with error ceiling","Statistical test tames LLM guessing in multiple choice","LLM answer sets with provable risk via p-values","P-value method matches conformal guarantees for LLMs","Risk-controlled answer sets from LLM responses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000142,"raw_usage":{"total_tokens":1032,"prompt_tokens":801,"completion_tokens":231,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":170}},"tokens_in":545,"tokens_out":231,"duration_ms":2672,"temperature":1.0,"reasoning_tokens":170,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T23:14:51.522355+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a calibration-test split where calibration is confined to one set of subjects (e.g., STEM) and evaluation to another (e.g., humanities) with the same p-value rule; if the empirical miscoverage rate exceeds α beyond bootstrap sampling error, the exchangeability premise is violated and the claimed guarantee does not hold. A second check: fix calibration data and change only the decoding temperature or prompt template at test time, then observe whether coverage stays above $1-\\alpha$.","supporting_citations":[{"cited_title":"SC on U : Selective conformal uncertainty in large language models","cited_arxiv_id":null,"evidence_quote":"Defines the hypothesized synergy between split conformal prediction and significance testing that the proposed method claims to realize."},{"cited_title":"Benchmarking llms via uncertainty quantification","cited_arxiv_id":null,"evidence_quote":"Points to the MCQA task setting in which the framework is instantiated and evaluated."}],"review_version":1}