{"id":"f8896cd3-8ff7-4c4d-90a1-3384289f5bb6","arxiv_id":"2608.07827","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Sequence-level token-probability averaging yields calibrated confidence for math question answering, and appending the verification prompt to the original generation matches standard re-prompting at 88% lower token-processing cost.","lead":"This paper compares ways to estimate how confident a large language model is that its math answer is correct, using the token probabilities the model already produces. It finds that averaging probabilities across the full reasoning path gives well-calibrated confidence, and that a cheap in-situ verification trick matches a more expensive one while cutting processing cost by 88%.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Regex-based correctness labels are the load-bearing assumption: unreported extraction failures likely concentrate in low-confidence outputs and could drive the ECE rankings behind Findings 1 and 3; a manual-label robustness check is required.","rationale":"The reader's weakest_assumption identifies the correctness-label extraction as the load-bearing assumption, and my read converges on the same point. All ECE, MCE, and Brier numbers in the paper are computed against regex-derived labels; the Limitations section explicitly acknowledges the associated extraction risk, but the paper does not quantify extraction failures or test whether they correlate with confidence. Because every headline comparison, including Finding 1 and Finding 3, is an empirical calibration comparison over those same labels, a systematic labeling artifact would directly alter the reported rankings. The mechanism is concrete: low-confidence outputs are more likely to be malformed or to deviate from the '####' template, so regex failures would disproportionately relabel correct-but-uncertain answers as incorrect, pushing empirical accuracy down in low-confidence bins and changing ECE for all estimators. Sequence-based and answer-based estimators use different token subsets and aggregations, so they would not be affected identically, meaning the central 'sequence beats answer' result could be an artifact. In-situ p(True) and standard p(True) share the same correctness labels, so Finding 3 is exposed as well. Other concerns, such as cBALD's per-pair dropout tuning or the absence of confidence intervals in the main tables, are secondary: the tuning is disclosed and the appendix's bootstrap intervals show the main sequence-versus-answer gaps are mostly non-overlapping. The proposed manual-label test is directly feasible and would settle whether the concern lands. If the test shows stable rankings under corrected labels, the central claims are well supported; if not, the headline findings need to be revisited. Accordingly, I agree with the reader and leave the verdict unchanged at CONDITIONAL.","tokens_in":27543,"tokens_out":9451,"duration_ms":105484,"concrete_test":"Take a random sample of 200-300 outputs per model-dataset pair from the evaluation splits, with emphasis on GSMHard and DeepSeek-R1-8B where format adherence is weakest. Manually annotate each output's correctness using a more robust protocol (parse \\boxed{} or the last numeric expression when '####' is absent, or use an LLM judge on ambiguous cases) and compare against the regex labels. Report the extraction failure rate by confidence bin, then recompute ECE for SEQAVG, ANSAVG, and in-situ p(True) using the corrected labels. If the sequence-versus-answer ordering and the in-situ-versus-p(True) ECE differences persist within bootstrap intervals, the concern is resolved and the central claims stand.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Every headline result in Sections 6.2 and 6.3 is an ECE comparison computed against binary correctness labels produced by regex extraction of the final answer after the '####' delimiter (plus \\boxed{} for DeepSeek-R1-8B), as described in Appendix C. The paper itself concedes in the Limitations that 'regex-based answer extraction using a fixed delimiter' can introduce extraction errors for 'malformed or non-conforming outputs,' but it never reports the extraction success rate, the failure modes, or the correlation between extraction failures and confidence.\n\nThe risk is not merely label noise. If malformed outputs are more frequent when the model is uncertain, then low-confidence correct answers are systematically relabeled as incorrect. That shifts the empirical accuracy of the low-confidence bins downward and changes ECE for every estimator. Because sequence-based estimators aggregate probabilities over the entire reasoning trajectory while answer-based estimators use only the extracted answer tokens, their confidence scores respond differently to a missing or malformed '####' block. The consistent 'sequence beats answer' ordering in Table 2 could therefore be an artifact of extraction failures rather than genuine trajectory-level signal. The same labels also underlie Finding 3 (in-situ versus standard p(True)) and the post-hoc calibration transfer analysis, so a single labeling artifact propagates across all central claims.\n\nThis is the weakest point of the paper's empirical foundation. Other concerns, such as per-pair dropout tuning for cBALD or missing main-table confidence intervals, are disclosed and do not infect every finding the same way.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents an empirical study of confidence estimation and calibration for large language models on mathematical question answering. It compares single-pass estimators that reuse token probabilities (sequence vs. answer-only, arithmetic vs. geometric aggregation), multi-pass estimators (self-verification p(True), an in-situ variant, and Monte Carlo Dropout cBALD), and two post-hoc calibration methods (isotonic regression and Platt scaling) across five open-weight models and three math QA datasets. The main findings are: (1) sequence-level aggregation of token probabilities yields lower ECE than answer-only estimators in every model-dataset pair; (2) cBALD often lowers ECE relative to p(True) but requires per-pair dropout-rate tuning and multiple passes; (3) in-situ self-verification achieves calibration comparable to standard p(True) while reducing token-processing overhead by about 88%; and (4) post-hoc calibration is generally effective but its data efficiency and cross-domain transfer are asymmetric and depend on dataset difficulty.","tokens_in":27796,"tokens_out":4943,"duration_ms":49701,"significance":"If the reported results are robust, the paper offers practically useful guidance for LLM confidence estimation: full-trajectory probability aggregation is a cheap and competitive default, and in-situ verification is an efficient alternative to full re-prompting. The study spans a diverse set of open-weight models and datasets and includes bootstrap confidence intervals in the appendix, which strengthens confidence in the qualitative conclusions. The paper is also transparent about the tuning cost of MC Dropout and the limitations of regex-based answer extraction. The main caveat is that all headline claims rest on correctness labels derived from regex extraction of a fixed delimiter, and the paper does not quantify how extraction failures could affect the ECE comparisons.","major_comments":[{"comment":"The correctness labels used for every ECE, MCE, and Brier score are obtained by regex extraction of text after the '####' delimiter (plus \\boxed{} for DeepSeek-R1-8B). The paper never reports the extraction success rate, the handling of outputs without the delimiter, or the correlation between extraction failures and model confidence. Since sequence-based estimators and answer-based estimators weight the presence and content of the final answer differently, systematic extraction failures could differentially affect their confidence scores and distort the 'sequence beats answer' ordering in Table 2 and the in-situ p(True) comparisons in Table 11. A robustness check with manual or semantically-matched labels on a stratified subsample, or at least a report of per-model extraction success rates and a re-analysis excluding unparseable outputs, is necessary to support the central claims.","section":"Section 6.1, Appendix C, Limitations"},{"comment":"The comparison between cBALD and p(True) in Section 6.2 is not controlled for hyperparameter tuning. The dropout rate for cBALD is selected on a 100-example tuning subset for each model-dataset pair (Appendix B.2), whereas p(True) uses a fixed template with no tuned components. In many rows of Table 7b the cBALD and p(True) bootstrap confidence intervals overlap, so the statement in Finding 2 that 'cBALD often achieves lower calibration error than p(True)' overstates the evidence. Reporting an untuned cBALD (e.g., a fixed dropout rate) or equivalently tuning p(True)'s verification threshold would put the comparison on equal footing, and the claim should be qualified by the extent of CI overlap.","section":"Section 4.3, Table 2, Appendix B.2"}],"minor_comments":[{"comment":"The main tables report only point estimates without confidence intervals; because the appendix shows nontrivial overlap for several key comparisons, consider adding confidence intervals or at least indicating which differences are significant, or explicitly referring the reader to the appendix tables.","section":"Table 2 and Table 3"},{"comment":"For Llama-3.2-3B on GSM8K, isotonic regression increases p(True) ECE by 39% relative to the uncalibrated estimator. The text says both methods 'generally reduce' calibration error, which is technically accurate, but it would be helpful to note this exception explicitly in the prose.","section":"Section 6.3, Table 3"},{"comment":"The paper does not state whether outputs that fail the regex extraction are marked as incorrect, discarded, or assigned a default mask. Since this directly affects the correctness labels, adding one sentence describing the fallback behavior would address a central reproducibility question.","section":"Appendix C"},{"comment":"The first sentence of the abstract contains a grammatical break: 'Prior work has shown that token probabilities are often overconfident, we investigate whether...' should be split into two sentences or joined with a semicolon.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper is well within the scope of the journal and the empirical design is largely sound. The main obstacle to acceptance is the unquantified reliance on regex-based correctness labels; the authors should be required to report extraction success rates and run at least a partial manual-label robustness check on the central ECE comparisons before the paper is accepted. The cBALD-versus-p(True) comparison also needs a fairer tuning protocol or substantially more cautious wording."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a solid, well-organized empirical study with one genuinely new and useful variant (in-situ self-verification) and a clean headline finding (sequence-level token probability aggregation beats answer-only estimators), but every headline ECE number rests on regex-extracted correctness labels, and the paper never demonstrates that extraction failures don't correlate with confidence. That is the load-bearing assumption to check before relying on the rankings.\n\nWhat's new and good. The in-situ p(True) idea—appending the verification prompt to the original generation instead of re-encoding the full question–answer pair—is simple, practical, and gives comparable ECE at an average 88% token-processing reduction. That is a real contribution. The fixed-trajectory MC Dropout design, measuring variation in answer-token predictive distributions rather than regenerating reasoning paths, is a thoughtful way to make MC Dropout affordable. The systematic comparison across five models and three datasets, with bootstrap CIs and bin-count robustness in the appendix, is thorough. The finding that calibration data efficiency depends on dataset difficulty is also worth reporting.\n\nSoft spots. The label concern is real. The paper concedes in the Limitations that regex extraction introduces errors, but it never reports the extraction success rate, the failure modes, or whether failures correlate with confidence. For DeepSeek-R1-8B, the format mismatch is acknowledged and likely explains its degraded numbers; those rows at least need a caveat. More broadly, if malformed outputs are more frequent when the model is uncertain, low-confidence correct answers get relabeled as incorrect, which shifts ECE and could favor sequence-based estimators if they respond differently to a missing delimiter. The qualitative pattern is consistent across many model–dataset pairs, so random label noise would attenuate rather than create it, but the systematic-correlation story is plausible and unaddressed. A manual annotation of a few hundred outputs, or at least extraction-success-rate reporting, would settle it.\n\nSecondary issues are minor. cBALD's dropout rate is tuned per model–dataset on a 100-example split while p(True) is untuned, so the cBALD-vs-p(True) comparison is asymmetric—though disclosed, it still matters. Main tables omit CIs, but the appendix has them and they back the qualitative conclusions. No code is released yet, though the paper says it will be.\n\nWho this is for: anyone building confidence estimates or calibration pipelines for open-weight LLMs, especially on math QA. It is a practical, honest empirical reference. It deserves a serious referee: the claims are concrete, the design is transparent, and the label robustness issue is fixable rather than fatal.\n\nRecommendation: send to peer review, with a request for an extraction-error robustness check and a re-analysis that either handles or clearly excludes the DeepSeek rows.","headline":"Solid empirical paper on LLM confidence; the in-situ p(True) trick is worth knowing, but verify the regex-label robustness before trusting the ECE rankings.","tokens_in":28370,"tokens_out":4275,"would_cite":true,"duration_ms":43879,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Token probabilities aggregated across the full reasoning trajectory, not the answer tokens alone, produce well-calibrated confidence for mathematical question answering.","keywords":["token probabilities","confidence calibration","expected calibration error","mathematical reasoning","self-verification","Monte Carlo dropout","isotonic regression","in-situ verification"],"falsifier":"Recompute the ECE tables using a second, independent judge that does not rely on the '####' delimiter to label a random sample of each model's outputs, focusing on outputs the regex flags as malformed; if sequence-based estimators then lose their advantage over answer-only estimators, the central claim collapses.","tokens_in":27325,"feed_emoji":"📊","tokens_out":5340,"duration_ms":49559,"temperature":0.7,"pith_summary":"The paper asks whether the token probabilities an LLM already produces while solving math problems can yield trustworthy confidence estimates, despite being individually overconfident. It claims yes: aggregating small but consistent probability differences over the whole reasoning trajectory gives well-calibrated confidence, and this single-pass sequence aggregation beats answer-only estimators in every model–dataset pair tested, sometimes beating more expensive multi-pass verification. A cheap in-situ verification that appends the verification question to the original generation matches standard p(True) calibration while cutting token-processing overhead by 88%. Post-hoc isotonic regression and Platt scaling further reduce in-domain calibration error, with isotonic regression effective from about 50 examples. These results matter because they offer a practical route to calibrated confidence with little or no extra compute.","feed_headline":"Aggregate whole reasoning path for calibrated LLM confidence","feed_subtitle":"Averaging token probabilities over the full answer, not just final tokens, beats costlier verification in math QA tests.","key_machinery":"The load-bearing object is the sequence-level aggregation of per-token probabilities, instantiated as SEQAVG (arithmetic mean of token probabilities over the full generated sequence) and SEQJOINT (length-normalized joint probability, equivalent to the geometric mean). These estimators reuse the token probabilities already computed during generation, so they add no extra inference passes; the argument is that weak per-token signal accumulates across the reasoning trajectory. The multi-pass machinery is BALD computed over MC Dropout samples at answer-token positions, converted to a confidence score by a monotone decreasing map, and the in-situ p(True) variant that appends the verification prompt to the original generation so only the appended tokens need new encoding.","core_discovery":"The paper's central empirical claim is that calibrated confidence for mathematical question answering does not require additional forward passes. Sequence-level aggregation of token probabilities from the original generation, via arithmetic mean or length-normalized joint probability, achieves lower expected calibration error than answer-token-only estimators across all five models and three datasets, and in several settings lower error than the multi-pass p(True) and MC Dropout estimators. The explanation offered is that no single token is informative: correct responses tend to show rising average token probability along the trajectory while incorrect ones fall, so small deviations accumulate into a discriminative signal. The paper also claims that an in-situ verification variant that appends the verification query to the original generation matches p(True) calibration while reducing token-processing overhead by 88%, and that MC Dropout captures distributional uncertainty at answer-token positions, but only with per-model–dataset dropout-rate tuning.","pith_inferences":["If the trajectory-aggregation mechanism is general, the same mean- or joint-probability estimators should calibrate confidence on other verifiable reasoning tasks such as code generation or proof steps, a testable extension beyond math word problems.","The paper's local-window suggestion could be pushed further: logging where the moving average of token probability dips may locate the erroneous step, linking calibration with error localization.","The asymmetry result implies calibration-set sampling should be difficulty-weighted rather than uniform; a testable extension is to construct calibration sets enriched with incorrect examples and measure whether 50-example efficiency improves.","In-situ verification's 88% reduction suggests that verification can be fused into the original decoding state, so confidence could be computed on the fly during generation rather than after a second pass."],"forward_implications":["Sequence-based single-pass estimators become a strong default for confidence on math QA whenever token probabilities are accessible, avoiding extra inference cost.","In-situ self-verification provides calibrated confidence at about 88% lower token-processing overhead than standard p(True), a saving that grows with answer length.","Isotonic regression reduces calibration error substantially with as few as 50 labeled examples on several settings, making post-hoc calibration data-cheap in-domain.","Calibration mappings transfer asymmetrically across datasets and models, so calibrators should be fit on data matched to the target difficulty and model.","MC Dropout gives calibrated confidence on hard datasets but requires dropout-rate selection and roughly five forward passes, so it is only worth its cost when single-pass estimators underperform."],"supporting_citations":[{"why":"supplies the p(True) self-verification method and the token-index normalization the paper compares against and extends with its in-situ variant.","marker":"[Kadavath et al., 2022]"},{"why":"provides MC Dropout as an approximate Bayesian uncertainty estimator, the basis of the cBALD multi-pass estimator.","marker":"[Gal and Ghahramani, 2016]"},{"why":"defines calibration and the ECE/MCE metrics and the temperature-scaling baseline used in post-hoc calibration.","marker":"[Guo et al., 2017]"},{"why":"GSM8K is one of the three evaluation datasets and supplies the '####' final-answer format used for correctness labels.","marker":"[Cobbe et al., 2021]"},{"why":"SVAMP is the second benchmark; its difficulty contrast supports the dataset-difficulty calibration findings.","marker":"[Patel et al., 2021]"},{"why":"GSMHard is the harder benchmark and also contributes the joint-probability length-normalization convention.","marker":"[Gao et al., 2023]"},{"why":"Platt scaling is one of the two post-hoc calibration methods evaluated.","marker":"[Platt et al., 1999]"},{"why":"isotonic regression, the other post-hoc calibration method, is shown to work with as few as 50 examples.","marker":"[Zadrozny and Elkan, 2002]"},{"why":"BALD is the disagreement score at answer-token positions that defines the MC-Dropout confidence estimator.","marker":"[Houlsby et al., 2011]"}],"fun_headline_variants":["Sequence-level token averaging yields calibrated math QA confidence","No extra passes needed: average token probabilities for calibration","Aggregate token probabilities over full answer for calibrated confidence","Full answer token averaging beats verification for math QA calibration","Math QA confidence calibration from whole-sequence token probabilities"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"All correctness labels come from regex extraction of the final answer after the '####' delimiter (plus \\boxed{} for one model) and the one-shot demonstration anchors models to that format; if extraction errors are systematic and correlate with confidence, every ECE comparison would be distorted.","fun_headline_variants_meta":{"raw":{"variants":["Sequence-level token averaging yields calibrated math QA confidence","No extra passes needed: average token probabilities for calibration","Aggregate token probabilities over full answer for calibrated confidence","Full answer token averaging beats verification for math QA calibration","Math QA confidence calibration from whole-sequence token probabilities"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000271,"raw_usage":{"total_tokens":1617,"prompt_tokens":925,"completion_tokens":692,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":618}},"tokens_in":541,"tokens_out":692,"duration_ms":6881,"temperature":1.0,"reasoning_tokens":618,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:48:44.455701+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the ECE tables using a second, independent judge that does not rely on the '####' delimiter to label a random sample of each model's outputs, focusing on outputs the regex flags as malformed; if sequence-based estimators then lose their advantage over answer-only estimators, the central claim collapses.","supporting_citations":[],"review_version":1}