{"id":"e8cfe3aa-5f11-4199-959e-d823876ff534","arxiv_id":"2602.07842","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Existing confidence-calibration methods systematically understate LLM confidence on questions with multiple valid answers; a new benchmark and a sum-of-probabilities method (SCA) improve mixed-answer calibration.","lead":"This paper shows that LLM confidence scores drop as questions gain more correct answers, even though the models answer more of them correctly. It introduces a 12,000-question benchmark, MACE, and a simple fix, SCA, that sums confidence across several sampled answers rather than trusting the most confident one.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MACE Math-domain construction is internally inconsistent; ground-truth cardinalities may be uncontrolled, undermining the core cardinality and SCA evaluation.","rationale":"The reader identified this same issue as the weakest assumption, and I concur; the paper's central claims require that MACE answer sets are complete and exactly 1/2/4/6. The Math construction text is self-contradictory: Table 5 gives an interval template, but §A.5 says queries are generated by uniformly sampling numbers from [0,1,000,000] without constructing natural questions or fixing answer counts. Even if the interval template is the intended method, the appendix does not describe how lower_num and upper_num are chosen to yield exactly 1, 2, 4, or 6 numbers of a given type, and the outlined sampling procedure cannot guarantee those counts. Because the cardinality labels drive every analysis (e.g., Table 3, Figure 3, Figure 4, Table 4), this is load-bearing. The AUROC-only evaluation of SCA is a separate weakness, but it is secondary: even a perfect calibration metric would not rescue a benchmark with mislabeled answer counts. If the Math domain is re-generated or excluded and the trends persist, the paper's conclusions are strengthened; if not, the headline is an artifact. The reader's CONDITIONAL verdict remains appropriate pending this check.","tokens_in":22221,"tokens_out":4432,"duration_ms":42851,"concrete_test":"Reconstruct the Math domain as described: for each QA pair, take the interval bounds implied by the template (or, if the sampling description is literal, generate intervals from the sampled numbers) and compute the exact number of valid answers (e.g., count primes in (lower_num, upper_num)). Compare to labeled cardinality. If a substantial fraction (say >5%) of 4a/6a questions do not have exactly 4/6 valid answers, the Math domain must be excluded or re-generated. As a second independent check, rerun the full analysis on the five non-Math domains alone; if the monotone accuracy/confidence trend and SCA's AUROC advantage persist without Math, the core phenomenon is not purely an artifact, but if they weaken substantially, the headline claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The MACE benchmark's ground-truth answer counts are the linchpin of both empirical claims (accuracy rises, confidence falls with cardinality; SCA improves calibration). The construction of the Math domain is internally inconsistent: Table 5 describes an interval template (\"Name one {number type} number between {lower_num} and {upper_num}\") that could in principle be used to target 1/2/4/6 answers by choosing interval bounds, but Appendix §A.5 says \"we do not construct natural questions explicitly\" and instead \"generate queries by uniformly sampling numbers from the interval [0,1,000,000]\". Uniformly sampling a number is not the same as constructing an interval around it, and the interval [0,1e6] would contain vastly more than 1, 2, 4, or 6 primes/squares/cubes/fibonacci/triangular numbers. No rule is given for mapping sampled numbers to intervals that yield the labeled cardinalities; therefore the 4a and 6a Math questions may have far more valid answers than labeled (or uncontrolled counts). If so, the observed accuracy increase with \"cardinality\" and the confidence decrease could be due to question difficulty or label noise, and SCA's AUROC gains could be an artifact of noisy labels rather than genuine multi-answer calibration. Because every other domain's construction also relies on Wikidata completeness and manual verification, the same completeness risk applies, but Math is the one domain where the text is internally contradictory and cannot be reconstructed. This is not a stylistic issue: the entire cardinality sweep and SCA evaluation depend on exact answer set counts, and §3.5 claims the construction guarantees them.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces MACE, a 12,000-question benchmark with 1/2/4/6 correct answers across six factual domains, and uses it to evaluate 15 training-free confidence calibration methods on four LLM families. The main empirical claim is that as the number of valid answers increases, QA accuracy rises while estimated confidence falls, causing miscalibration on mixed answer-count questions. The paper proposes Semantic Confidence Aggregation (SCA), which sums token-level generation probabilities over sampled responses (optionally filtered by a threshold), and reports that SCA achieves the best AUROC on mixed-answer settings while remaining competitive on single-answer questions.","tokens_in":22607,"tokens_out":9327,"duration_ms":91904,"significance":"The paper targets a real and understudied problem: existing calibration methods are designed for single-answer QA, and multi-answer questions create a systematic mismatch between consistency-based uncertainty and correctness. The trend documented in Table 3 is nontrivial and potentially useful; the benchmark, if carefully constructed, would be a valuable resource. A particular strength is that the main SCA result at tau=0 has no fitted parameters, so the comparison is not a tuning artifact; the authors also include a wide set of baselines and model families. However, the current evidence is not yet sufficient for the paper's strongest claims: the Math domain construction is internally inconsistent, and SCA is evaluated only by AUROC, not by probability-scale calibration, so the 'state-of-the-art calibration' claim is not demonstrated.","major_comments":[{"comment":"The Math-domain construction is internally inconsistent and undermines the cardinality control that drives both main results. Section 3.4 and Table 5 promise questions of the form 'Name one {number type} number between {lower_num} and {upper_num}' with exactly 1, 2, 4, or 6 correct answers, but Appendix A.5 states that 'we do not construct natural questions explicitly' and instead 'generate queries by uniformly sampling numbers from the interval [0,1,000,000]'. No rule is given for converting a sampled number into an interval containing exactly the labeled number of primes/squares/cubes/Fibonacci/triangular numbers; a random interval in [0,10^6] will typically contain far more than 6 such numbers. As written, the Math answer-count labels are not reproducible, so the observed accuracy/confidence trends and SCA gains may be driven by label noise or uncontrolled difficulty. Please provide t","section":"Appendix A.5 vs. Table 5 / Section 3.4"},{"comment":"SCA is claimed to 'achieve state-of-the-art calibration performance', but the paper reports only AUROC for SCA. AUROC is a rank-based discrimination metric and is invariant to monotone transformations of the confidence score; it does not measure whether confidence equals accuracy on the probability scale. Moreover, Eq. (2) sums absolute token-level probabilities over sampled responses and is not normalized to [0,1]; depending on N and duplicate responses, the score can exceed 1. No ECE or reliability diagram is reported for SCA in Table 18 (which lists ECE only for the 15 baselines). Please add ECE/reliability analysis for SCA (and ideally for the other models) and specify how the raw sum is rescaled; otherwise the central contribution is a ranking heuristic, not a calibrated confidence.","section":"Section 6.3, Table 4, Table 18, Eq. (1)-(2)"},{"comment":"The SCA advantage over existing methods is small and is not accompanied by significance testing. On LLaMA-3.1-70B, SCA(tau=0) AUROC is 80.5 vs. 80.9 for Perplexity on [1], 81.2 vs. 80.3 on [1,2], 79.2 vs. 78.1 on [1,2,4], and 76.7 vs. 75.3 on [1,2,4,6]; on Qwen-2.5-72B the pattern is similar. Thus on the single-answer setting SCA is actually below the best baseline, and the mixed-setting gains are within 1-2 AUROC points. No confidence intervals, multiple runs, or paired significance tests are reported, and SCA is evaluated on only two of the model families used elsewhere. Please provide variability estimates and test the mixed-setting advantage against the best baseline.","section":"Section 6.3, Table 4, Appendix D.1"},{"comment":"The central trend that 'estimated confidence decreases' requires a common confidence scale, but the paper never defines how the entropy/perplexity quantities in Appendix B are converted to the 'Confidence' percentages in Tables 3 and 8-15. Equations (3), (4), (7), and (12) define uncertainties U, not confidences C; without the transformation, the reported 1a->6a declines for Prob Entropy, N-Prob Entropy, Sem Entropy, and Perplexity cannot be interpreted. Please state the explicit monotone mapping used (or, if raw values are reported, correct the tables), since this mapping is load-bearing for the paper's first finding.","section":"Section 4.1, Appendix B, Tables 3/8-15"}],"minor_comments":[{"comment":"Typo: 'exisiting' should be 'existing'.","section":"Section 5.2"},{"comment":"The x-axis labels '[1]', '[1,2]', etc. are not defined in the caption. Please explain that they denote mixed sets of answer counts.","section":"Figure 3"},{"comment":"The placeholders '/githubCode' and 'Datasets' appear without URLs. Please provide the actual code and data links.","section":"Section 1 / Dataset links"},{"comment":"Several references are malformed or incomplete (e.g., 'Joue-An Hu and 1 others', 'Hurst et al.' with truncated author lists). Use a consistent, complete reference format.","section":"References"},{"comment":"The text says results for other models are reported in Appendix D.1, but only Qwen-2.5-72B is actually shown. Clarify which models were tested and where the remaining results are.","section":"Appendix D.1"},{"comment":"Minor formatting issues: 'over70%' should read 'over 70%'; also the threshold discussion would benefit from a reference to the figure or table that shows the cluster probability distribution.","section":"Appendix C.2"}],"recommendation":"major_revision","confidential_remarks":"The Math-construction inconsistency is the main blocker to acceptance. I would ask the authors to either supply the exact interval-generation algorithm and verify cardinalities, or drop Math from the headline results. In addition, the SCA claim needs probability-scale calibration evidence (ECE/reliability) and a clear normalization of the raw sum; as it stands, the method is a ranking heuristic. These are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nWhat you should know first: the paper's central observation—LLMs get more accurate but less confident as the number of valid answers grows—is clearly documented across many tables and models, and it's a point worth taking seriously. The MACE benchmark is a real contribution, even if it ships with one ugly seam.\n\nThe genuinely new thing is the systematic study: a 12k-question benchmark across six domains with 1/2/4/6 ground-truth answers, 15 training-free calibration methods, four model families. The finding that consistency-based methods over-penalize multi-answer questions, and that larger models show a sharper confidence drop (because they know more valid answers), is well supported by the knowledge-coverage analysis. SCA itself is simple—sum token-level probabilities of sampled responses—but at τ=0 it does beat the baselines on AUROC in mixed settings, and the fact that it needs no tuning is nice.\n\nNow the soft spots, in order of importance.\n\nFirst, SCA's \"state-of-the-art calibration\" claim rests entirely on AUROC. AUROC is a discrimination metric, not a calibration metric. Table 18 reports ECE for every baseline but SCA is absent. That's a real gap; the method may simply produce scores that rank well, and we have no evidence it is calibrated on a probability scale. This is fixable—normalize the sum and report ECE—but it is load-bearing.\n\nSecond, the Math domain construction is internally inconsistent. Appendix A.5 says queries are generated by uniformly sampling numbers from [0,1,000,000], but Table 5's template asks for \"a number between lower and upper,\" and §3.5 claims exact answer counts are guaranteed. No rule connects the sampled number to an interval that yields exactly 1, 2, 4, or 6 primes/squares/cubes/etc. If Math cardinalities are uncontrolled, the cardinality sweep and SCA evaluation have an unmeasured label-noise problem in one of six domains. The other five domains rest on Wikidata plus manual verification (κ=0.94), so I'd expect the trend to survive, but the paper needs to reconcile the appendix or the benchmark can't be reconstructed.\n\nThird, the conclusion says \"all these methods collapse\" under mixing, but the ECE table shows several baselines improving (e.g., Prob Entropy 26.2→19.4). The overstatement is minor, but it undersells a more nuanced story.\n\nFourth, the benchmark is not released, so the empirical claims can't be independently verified yet.\n\nSo: the paper deserves a serious referee. The central phenomenon is likely real, the benchmark is useful, and the flaws are addressable. I'd send it to review with a request for ECE on SCA, a rewritten Math-construction section, and a release commitment. For a reading group it's a solid choice for anyone working on calibration.","headline":"Useful benchmark and a plausible cardinality effect, but SCA's calibration claim needs ECE and the Math domain construction is broken.","tokens_in":23068,"tokens_out":3492,"would_cite":true,"duration_ms":34988,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Summing response probabilities fixes LLM confidence calibration","keywords":["confidence calibration","large language models","multiple correct answers","semantic consistency","token-level probabilities","benchmark","MACE","semantic confidence aggregation"],"falsifier":"Compute confidence on a new set of questions where answer cardinality is known by construction (e.g., generated from a database with explicit answer sets) and check whether the accuracy/confidence divergence and SCA's advantage persist; alternatively, re-annotate a random sample of MACE to see whether any 1a/2a/4a/6a questions actually have a different number of valid answers, and if the mismatch is large enough, the reported trends would disappear.","tokens_in":22133,"feed_emoji":"🎯","tokens_out":4852,"duration_ms":43399,"temperature":0.7,"pith_summary":"Large language models' confidence estimates are usually calibrated on questions with exactly one right answer. This paper shows that when a question has several correct answers, almost all existing training-free calibration methods systematically underestimate confidence: the model knows the answer but gives different valid answers across samples, and consistency-based methods read that disagreement as uncertainty. The paper builds a benchmark (MACE) with 12,000 questions that have 1, 2, 4, or 6 verified correct answers, and measures 15 calibration methods across four LLM families. It reports a paradox: accuracy rises with answer count while confidence falls, so in realistic mixed settings calibration collapses. The proposed fix, SCA, sums token-level probabilities over multiple sampled responses instead of trusting the single most confident response, and the paper reports that this restores state-of-the-art calibration in mixed settings while keeping single-answer behavior.","feed_headline":"Summing response probabilities fixes LLM confidence calibration","feed_subtitle":"A 12,000-question benchmark shows all 15 tested methods misread multiple valid answers as uncertainty; SCA restores alignment.","key_machinery":"SCA (Semantic Confidence Aggregation): sample N responses, optionally group them into semantic clusters, and take the question's confidence as the sum of token-level sequence generation probabilities over all clusters (or all responses when threshold τ=0). The key move is aggregating absolute probabilities rather than normalizing or counting agreement; absolute token probabilities already assign low weight to spurious answers, so the total reflects the model's belief across all valid answers instead of punishing disagreement. The other machinery is MACE, a benchmark built from Wikidata triplets and rule-synthesized math questions with exactly 1, 2, 4, or 6 ground-truth answers, which makes t","core_discovery":"The central discovery is that consistency-based confidence estimates conflate answer-space diversity with model uncertainty. On questions with multiple valid answers, a model that samples several different but equally correct answers appears to 'disagree with itself,' producing low confidence scores indistinguishable from genuine doubt. The paper shows this failure extends beyond consistency methods to token-probability and verbalized methods, and that larger models suffer sharper confidence drops because they know a wider set of correct answers and therefore alternate among more of them. The correction proposed is SCA: sum the per-token generation probabilities of all sampled responses (opt","pith_inferences":["The same accuracy–confidence divergence should appear in other 'multiple acceptable output' settings, such as open-ended QA, code generation, and summarization; a testable extension is whether SCA-style aggregation improves calibration for tasks with a set of acceptable outputs rather than a single reference.","The threshold-free result suggests that token-level probability mass is already concentrated on 'the set of things the model is willing to say'; summing absolute probabilities effectively computes a lower bound on the model's total belief over its answer space. One could test whether a closed-form Bayesian posterior over answer clusters reproduces SCA without sampling.","If answer sets in MACE are incomplete, the observed accuracy-increase/confidence-decrease pattern would be confounded by label noise; a stress test that re-annotates a random subset and measures robustness would clarify how much of the effect is real.","The 'larger models drop more' result implies that scaling up a model can worsen calibration on open questions even as accuracy improves; a practical consequence is that calibration targets should be conditioned on expected answer-set size, not just on question type."],"forward_implications":["In real-world deployments where questions can have several valid answers, existing confidence scores (consistency, verbalized, token-probability) systematically under-report certainty, which can mislead hallucination detection and retrieval-augmentation triggers.","Larger models are more susceptible to this underestimation because they know more correct answers; calibration comparisons between model sizes that ignore answer cardinality will be biased.","Simple summation of token-level probabilities across samples gives strong calibration without clustering or threshold tuning, meaning the correction is cheap to compute.","SCA preserves single-answer calibration while fixing mixed-answer settings, so it can replace existing training-free estimators without sacrificing previous gains.","Confidence on multi-answer questions should be interpreted as a sum over valid alternatives, not as the probability of one specific answer."],"fun_headline_variants":["Summing answer probabilities fixes LLM confidence drop","LLM confidence plummets when many answers are correct","Multi-answer questions break LLM confidence calibration","New method SCA recalibrates LLM confidence for multi-answer questions","LLMs misread multiple correct answers as uncertainty"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The MACE benchmark's ground-truth answer sets are complete and exactly 1, 2, 4, or 6 correct answers; if any answer list is missing valid answers—especially in the Math domain, whose construction is described inconsistently (uniform sampling vs interval templates)—the measured decline in confidence and the gains from SCA could be artifacts rather than properties of multi-answer questions.","fun_headline_variants_meta":{"raw":{"variants":["Summing answer probabilities fixes LLM confidence drop","LLM confidence plummets when many answers are correct","Multi-answer questions break LLM confidence calibration","New method SCA recalibrates LLM confidence for multi-answer questions","LLMs misread multiple correct answers as uncertainty"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000804,"raw_usage":{"total_tokens":3335,"prompt_tokens":674,"completion_tokens":2661,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":418,"completion_tokens_details":{"reasoning_tokens":2582}},"tokens_in":418,"tokens_out":2661,"duration_ms":17410,"temperature":1.0,"reasoning_tokens":2582,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T03:27:28.449186+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute confidence on a new set of questions where answer cardinality is known by construction (e.g., generated from a database with explicit answer sets) and check whether the accuracy/confidence divergence and SCA's advantage persist; alternatively, re-annotate a random sample of MACE to see whether any 1a/2a/4a/6a questions actually have a different number of valid answers, and if the mismatch is large enough, the reported trends would disappear.","supporting_citations":[],"review_version":1}