Pith. sign in

REVIEW 2 major objections 4 minor 87 references

From token probabilities to calibrated confidence: An empirical study of mathematical question answering

T0 review · 2 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Token probabilities aggregated across the full reasoning trajectory, not the answer tokens alone, produce well-calibrated confidence for mathematical question answering.

desk verdict Solid empirical paper on LLM confidence; the in-situ p(True) trick is worth knowing, but verify the regex-label robustness before trusting the ECE rankings. read the letter →

arxiv 2608.07827 v1 pith:PUJGITPD submitted 2026-08-08 cs.LG cs.CL

Avery Ma , Lorne Schell , Vin Bhaskara , Leila Pishdad This is my paper · ORCID
classification cs.LGcs.CL
keywords tokenprobabilitiesconfidencecalibrationexpectederrormathematicalreasoningself-verificationMonteCarlodropoutisotonicregressionin-situverification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether the token probabilities an LLM already produces while solving math problems can yield trustworthy confidence estimates, despite being individually overconfident. It claims yes: aggregating small but consistent probability differences over the whole reasoning trajectory gives well-calibrated confidence, and this single-pass sequence aggregation beats answer-only estimators in every model–dataset pair tested, sometimes beating more expensive multi-pass verification. A cheap in-situ verification that appends the verification question to the original generation matches standard p(True) calibration while cutting token-processing overhead by 88%. Post-hoc isotonic regression and Platt scaling further reduce in-domain calibration error, with isotonic regression effective from about 50 examples. These results matter because they offer a practical route to calibrated confidence with little or no extra compute.

What carries the argument

The load-bearing object is the sequence-level aggregation of per-token probabilities, instantiated as SEQAVG (arithmetic mean of token probabilities over the full generated sequence) and SEQJOINT (length-normalized joint probability, equivalent to the geometric mean). These estimators reuse the token probabilities already computed during generation, so they add no extra inference passes; the argument is that weak per-token signal accumulates across the reasoning trajectory. The multi-pass machinery is BALD computed over MC Dropout samples at answer-token positions, converted to a confidence score by a monotone decreasing map, and the in-situ p(True) variant that appends the verification prompt to the original generation so only the appended tokens need new encoding.

What would settle it

Recompute the ECE tables using a second, independent judge that does not rely on the '####' delimiter to label a random sample of each model's outputs, focusing on outputs the regex flags as malformed; if sequence-based estimators then lose their advantage over answer-only estimators, the central claim collapses.

Watch

Extended reading notes

Core claim

The paper's central empirical claim is that calibrated confidence for mathematical question answering does not require additional forward passes. Sequence-level aggregation of token probabilities from the original generation, via arithmetic mean or length-normalized joint probability, achieves lower expected calibration error than answer-token-only estimators across all five models and three datasets, and in several settings lower error than the multi-pass p(True) and MC Dropout estimators. The explanation offered is that no single token is informative: correct responses tend to show rising average token probability along the trajectory while incorrect ones fall, so small deviations accumulate into a discriminative signal. The paper also claims that an in-situ verification variant that appends the verification query to the original generation matches p(True) calibration while reducing token-processing overhead by 88%, and that MC Dropout captures distributional uncertainty at answer-token positions, but only with per-model–dataset dropout-rate tuning.

Load-bearing premise

All correctness labels come from regex extraction of the final answer after the '####' delimiter (plus \boxed{} for one model) and the one-shot demonstration anchors models to that format; if extraction errors are systematic and correlate with confidence, every ECE comparison would be distorted.

Editorial extensions

If this is right

  • Sequence-based single-pass estimators become a strong default for confidence on math QA whenever token probabilities are accessible, avoiding extra inference cost.
  • In-situ self-verification provides calibrated confidence at about 88% lower token-processing overhead than standard p(True), a saving that grows with answer length.
  • Isotonic regression reduces calibration error substantially with as few as 50 labeled examples on several settings, making post-hoc calibration data-cheap in-domain.
  • Calibration mappings transfer asymmetrically across datasets and models, so calibrators should be fit on data matched to the target difficulty and model.
  • MC Dropout gives calibrated confidence on hard datasets but requires dropout-rate selection and roughly five forward passes, so it is only worth its cost when single-pass estimators underperform.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the trajectory-aggregation mechanism is general, the same mean- or joint-probability estimators should calibrate confidence on other verifiable reasoning tasks such as code generation or proof steps, a testable extension beyond math word problems.
  • The paper's local-window suggestion could be pushed further: logging where the moving average of token probability dips may locate the erroneous step, linking calibration with error localization.
  • The asymmetry result implies calibration-set sampling should be difficulty-weighted rather than uniform; a testable extension is to construct calibration sets enriched with incorrect examples and measure whether 50-example efficiency improves.
  • In-situ verification's 88% reduction suggests that verification can be fused into the original decoding state, so confidence could be computed on the fly during generation rather than after a second pass.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. This paper presents an empirical study of confidence estimation and calibration for large language models on mathematical question answering. It compares single-pass estimators that reuse token probabilities (sequence vs. answer-only, arithmetic vs. geometric aggregation), multi-pass estimators (self-verification p(True), an in-situ variant, and Monte Carlo Dropout cBALD), and two post-hoc calibration methods (isotonic regression and Platt scaling) across five open-weight models and three math QA datasets. The main findings are: (1) sequence-level aggregation of token probabilities yields lower ECE than answer-only estimators in every model-dataset pair; (2) cBALD often lowers ECE relative to p(True) but requires per-pair dropout-rate tuning and multiple passes; (3) in-situ self-verification achieves calibration comparable to standard p(True) while reducing token-processing overhead by about 88%; and (4) post-hoc calibration is generally effective but its data efficiency and cross-domain transfer are asymmetric and depend on dataset difficulty.

Significance. If the reported results are robust, the paper offers practically useful guidance for LLM confidence estimation: full-trajectory probability aggregation is a cheap and competitive default, and in-situ verification is an efficient alternative to full re-prompting. The study spans a diverse set of open-weight models and datasets and includes bootstrap confidence intervals in the appendix, which strengthens confidence in the qualitative conclusions. The paper is also transparent about the tuning cost of MC Dropout and the limitations of regex-based answer extraction. The main caveat is that all headline claims rest on correctness labels derived from regex extraction of a fixed delimiter, and the paper does not quantify how extraction failures could affect the ECE comparisons.

major comments (2)
  1. [Section 6.1, Appendix C, Limitations] The correctness labels used for every ECE, MCE, and Brier score are obtained by regex extraction of text after the '####' delimiter (plus \boxed{} for DeepSeek-R1-8B). The paper never reports the extraction success rate, the handling of outputs without the delimiter, or the correlation between extraction failures and model confidence. Since sequence-based estimators and answer-based estimators weight the presence and content of the final answer differently, systematic extraction failures could differentially affect their confidence scores and distort the 'sequence beats answer' ordering in Table 2 and the in-situ p(True) comparisons in Table 11. A robustness check with manual or semantically-matched labels on a stratified subsample, or at least a report of per-model extraction success rates and a re-analysis excluding unparseable outputs, is necessary to support the central claims.
  2. [Section 4.3, Table 2, Appendix B.2] The comparison between cBALD and p(True) in Section 6.2 is not controlled for hyperparameter tuning. The dropout rate for cBALD is selected on a 100-example tuning subset for each model-dataset pair (Appendix B.2), whereas p(True) uses a fixed template with no tuned components. In many rows of Table 7b the cBALD and p(True) bootstrap confidence intervals overlap, so the statement in Finding 2 that 'cBALD often achieves lower calibration error than p(True)' overstates the evidence. Reporting an untuned cBALD (e.g., a fixed dropout rate) or equivalently tuning p(True)'s verification threshold would put the comparison on equal footing, and the claim should be qualified by the extent of CI overlap.
minor comments (4)
  1. [Table 2 and Table 3] The main tables report only point estimates without confidence intervals; because the appendix shows nontrivial overlap for several key comparisons, consider adding confidence intervals or at least indicating which differences are significant, or explicitly referring the reader to the appendix tables.
  2. [Section 6.3, Table 3] For Llama-3.2-3B on GSM8K, isotonic regression increases p(True) ECE by 39% relative to the uncalibrated estimator. The text says both methods 'generally reduce' calibration error, which is technically accurate, but it would be helpful to note this exception explicitly in the prose.
  3. [Appendix C] The paper does not state whether outputs that fail the regex extraction are marked as incorrect, discarded, or assigned a default mask. Since this directly affects the correctness labels, adding one sentence describing the fallback behavior would address a central reproducibility question.
  4. [Abstract] The first sentence of the abstract contains a grammatical break: 'Prior work has shown that token probabilities are often overconfident, we investigate whether...' should be split into two sentences or joined with a semicolon.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical comparisons use held-out test splits, and the only author-overlapping artifact (the BAAL library) is an implementation tool, not a load-bearing premise.

full rationale

The paper's central comparisons (Findings 1, 2, 3 and the post-hoc calibration analysis) are evaluated on held-out test splits disjoint from any fitting: confidence estimators are computed from token probabilities without training; the post-hoc calibrators (isotonic regression, Platt scaling) are fit on calibration splits (GSM8K/SVAMP training splits, 70% of GSMHard) and evaluated on test sets; ECE/MCE/Brier are computed against correctness labels. The dropout rate for cBALD is selected on a 100-example tuning subset per model-dataset pair and then fixed on the disjoint evaluation split; this is hyperparameter selection rather than fitting the reported ECE, and the paper discloses the tuning cost. The BAAL library (Atighehchian et al., 2022) cited for BALD computation shares an author, but it is an open-source implementation tool, not a load-bearing theoretical premise; the BALD definition itself is attributed to Houlsby et al. (2011) and computed from the models' predictive distributions. The regex-based answer extraction is a correctness-labeling risk acknowledged in the Limitations ("Correctness evaluation relies on regex-based answer extraction using a fixed delimiter; malformed or non-conforming outputs may therefore introduce extraction errors"), but this is a measurement-validity concern, not a circular step: the paper does not define correctness in terms of the confidence estimators or fit any estimator to the test labels. No equation in the paper defines a predicted quantity as equal by construction to an input quantity. No load-bearing argument reduces to a self-citation chain. The in-situ p(True) result is an empirical comparison, and the 88% token-processing reduction is computed from prompt token counts after the fact, not used to force the ECE comparison.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claims rest on standard LLM inference and dataset conventions rather than on new theoretical postulates. The main fitted inputs are the MC Dropout rate, selected per model-dataset pair on a tuning subset, and the hand-chosen BALD normalization; both affect the reported cBALD results. No new physical or model-level entities are introduced.

free parameters (3)
  • MC Dropout rate = per model-dataset pair from {0.10, 0.15, 0.20}, e.g., Llama-3.1-8B: 0.10 (GSM8K), 0.20 (GSMHard), 0.10 (SVAMP)
    Selected by grid search on a 100-example tuning subset to minimize ECE (Table 5); the evaluation split ECE for cBALD in Table 2 uses this selected rate. This per-pair tuning gives cBALD an advantage over untuned p(True) and is a fitted input to the headline comparison.
  • BALD-to-confidence normalization phi = clip(1 - BALD, 0, 1)
    Chosen by hand after trying top-k and top-p normalizations (Appendix B.1); the scale of BALD depends on the dropout rate and support, so this choice directly affects cBALD calibration and adds sensitivity.
  • Number of MC Dropout passes T = not stated in the main text; Figure 1(b) suggests around 5
    The paper reports that predictive entropy stabilizes after roughly five samples, but the exact T used to produce cBALD results in Table 2 is not given, making the reported cBALD ECE not fully reproducible from the text.
assumptions (5)
  • domain assumption Token probabilities from open-weight models are accessible and faithful next-token distributions at inference time.
    All single-pass estimators in Section 4.2 read P(y_i | x, y_<i) from the model; this is standard for open-weight LLMs.
  • domain assumption Regex extraction of the final answer after the '####' delimiter (or \boxed{} for DeepSeek-R1-8B) yields correct labels for answer correctness.
    Correctness function in Section 3.2 relies on extracted y_ans; Appendix C describes the extraction, and the Limitations section notes malformed outputs may introduce errors.
  • domain assumption Expected Calibration Error with 10 bins is a valid and comparable summary of calibration quality.
    Section 5 defines ECE/MCE following Guo et al. (2017); bin-count ablations in Table 12 show ECE is stable, but bin choice remains a modeling decision.
  • standard math MC Dropout with all supported dropout modules enabled approximates Bayesian inference over model parameters, so BALD is a meaningful uncertainty measure.
    Section 4.3, Eq. (1) uses the BALD decomposition from Houlsby et al. (2011) and the MC dropout approximation from Gal and Ghahramani (2016).
  • domain assumption The calibration, tuning, and evaluation splits are representative and independent draws from each dataset.
    Appendix C: GSMHard is randomly split 70/30; GSM8K and SVAMP use provided splits; calibration sets are reused for post-hoc fitting and dropout tuning, which is standard but assumes no leakage.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From token probabilities to calibrated confidence: An empirical study of mathematical question answering." pith.science (2026). https://pith.science/paper/PUJGITPD

@misc{pith2026260807827,
  author       = {Pith},
  title        = {Pith review of: From token probabilities to calibrated confidence: An empirical study of mathematical question answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PUJGITPD}},
  note         = {Machine review of arXiv:2608.07827}
}
read the original abstract

Confidence estimation for large language models (LLMs) aims to estimate the probability that a generated answer is correct, while calibration aligns these estimates with empirical accuracy. Prior work has shown that token probabilities are often overconfident, we investigate whether these readily available signals can nevertheless provide well-calibrated confidence estimation for mathematical question answering. We compare single-pass estimators, which reuse token probabilities from the original generation, with multi-pass estimators, which obtain additional confidence signals through verification or stochastic forward passes. While individual token probabilities can be highly saturated, we find that aggregating token probabilities over the full sequence captures small but consistent differences between correct and incorrect generations, yielding more informative confidence estimates. Multi-pass methods can yield calibrated confidence estimates. We study two such approaches: self-verification through re-prompting, including a lower-cost in-situ variant, and Monte Carlo Dropout, which derives confidence from variation across stochastic forward passes. We further evaluate two post-hoc calibration methods, Platt scaling and isotonic regression, both of which substantially reduce in-domain calibration error. However, their data efficiency varies with dataset difficulty, and the calibration mappings often transfer asymmetrically across datasets and models.

Figures

Figures reproduced from arXiv: 2608.07827 by the authors.

Figure 1
Figure 1. Confidence estimation analysis. (a) Moving average of token probabilities across the generated sequence. Confidence-relevant information arises from the accumulation of small token-level probability dif￾ferences across the reasoning trajectory. (b) BALD decomposition under MC Dropout. Predictive en￾tropy increases and stabilizes after roughly five samples, indicating that MC Dropout primarily captures distri￾butiona… view at source ↗
Figure 2
Figure 2. Post-hoc confidence calibration analysis. Reliability diagrams for Ans Avg: both (a) isotonic re￾gression and (b) Platt scaling improve calibration, with isotonic regression fitting a flexible monotonic mapping and Platt scaling applying a global parametric correction. (c) Effect of calibration set size on Ans Avg. Calibra￾tion error decreases quickly with calibration set size, with 50 examples already yielding subs… view at source ↗
Figure 3
Figure 3. Example input–output exchange on GSM8K with Llama-3.1-8B. The one-shot example is crucial in improving the success rate of our regex-based answer extraction. tel et al., 2021), three publicly available mathe￾matical question-answering benchmarks released for research use. Our BALD-score computation is adapted from the publicly available BAAL im￾plementation (Atighehchian et al., 2022), which is distributed under the… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Standard p(True) vs. in-situ p(True) on GSM8K with Llama-3.1-8B. Left: standard p(True) self￾verification via re-prompting, where the full question–answer pair is re-encoded. Right: in-situ p(True), where the verification prompt is appended directly to the original gen…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

87 extracted references · 66 canonical work pages

  1. [1]

    Attention is all you need , author=

  2. [2]

    Language models are few-shot learners , author=

  3. [3]

    arXiv preprint arXiv:2302.13971 , year=

    Llama: Open and efficient foundation language models , author=. arXiv preprint arXiv:2302.13971 , year=

    arXiv

  4. [4]

    Achiam, Josh and Adler, Steven and Agarwal, Sandhini and Ahmad, Lama and Akkaya, Ilge and Aleman, Florencia Leoni and Almeida, Diogo and Altenschmidt, Janko and Altman, Sam and Anadkat, Shyamal and others , journal=

  5. [5]

    arXiv preprint arXiv:2312.11805 , year=

    Gemini: a family of highly capable multimodal models , author=. arXiv preprint arXiv:2312.11805 , year=

    arXiv

  6. [6]

    arXiv preprint arXiv:2412.19437 , year=

    Deepseek-v3 technical report , author=. arXiv preprint arXiv:2412.19437 , year=

    arXiv

  7. [7]

    On calibration of modern neural networks , author=

  8. [8]

    arXiv preprint arXiv:2207.05221 , year=

    Language models (mostly) know what they know , author=. arXiv preprint arXiv:2207.05221 , year=

    arXiv

Show all 87 references
  1. [9]

    Atighehchian, Parmida and Branchaud-Charron, Frederic and Freyberg, Jan and Pardinas, Rafael and Schell, Lorne and Pearse, George , year=

  2. [10]

    Dropout as a bayesian approximation: Representing model uncertainty in deep learning , author=

  3. [11]

    Uncertainty in

    Gal, Yarin and others , year=. Uncertainty in

  4. [12]

    Wu, Shijie and Irsoy, Ozan and Lu, Steven and Dabravolski, Vadim and Dredze, Mark and Gehrmann, Sebastian and Kambadur, Prabhanjan and Rosenberg, David and Mann, Gideon , journal=

  5. [13]

    Harnessing

    Yu, Xinli and Chen, Zheng and Lu, Yanbin , booktitle=emnlp-industry, year=. Harnessing

  6. [14]

    Guo, Yue and Xu, Zian and Yang, Yi , booktitle=findings-emnlp, year=. Is

  7. [15]

    arXiv preprint arXiv:2305.15525 , year=

    Large language models are few-shot health learners , author=. arXiv preprint arXiv:2305.15525 , year=

    arXiv

  8. [16]

    Ravenda, Federico and Bahrainian, Seyed Ali and Raballo, Andrea and Mira, Antonietta and Kando, Noriko , booktitle=acl, year=. Are

  9. [17]

    Can Large Language Models Accurately Generate Answer Keys for Health-related Questions? , author=

  10. [18]

    Unilaw-r1: A large language model for legal reasoning with reinforcement learning and iterative inference , author=

  11. [19]

    Automating legal interpretation with

    Luo, Kangcheng and Huang, Quzhe and Jiang, Cong and Feng, Yansong , booktitle=acl, year=. Automating legal interpretation with

  12. [20]

    Enabling discriminative reasoning in

    Deng, Chenlong and Mao, Kelong and Zhang, Yuyao and Dou, Zhicheng , booktitle=findings-emnlp, year=. Enabling discriminative reasoning in

  13. [21]

    Mind the confidence gap: Overconfidence, calibration, and distractor effects in large language models , author=

  14. [22]

    Reducing conversational agents’ overconfidence through linguistic calibration , author=

  15. [23]

    Training language models to follow instructions with human feedback , author=

  16. [24]

    Learning to summarize with human feedback , author=

  17. [25]

    arXiv preprint arXiv:2110.14168 , year=

    Training verifiers to solve math word problems , author=. arXiv preprint arXiv:2110.14168 , year=

    arXiv

  18. [26]

    Gao, Luyu and Madaan, Aman and Zhou, Shuyan and Alon, Uri and Liu, Pengfei and Yang, Yiming and Callan, Jamie and Neubig, Graham , booktitle=icml, year=

  19. [27]

    Patel, Arkil and Bhattamishra, Satwik and Goyal, Navin , booktitle=naacl, year=. Are

  20. [28]

    Transforming classifier scores into accurate multiclass probability estimates , author=

  21. [29]

    Advances in Large Margin Classifiers , year=

    Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods , author=. Advances in Large Margin Classifiers , year=

  22. [30]

    Wikiqa: A challenge dataset for open-domain question answering , author=

  23. [31]

    Kantharaj, Shankar and Do, Xuan Long and Leong, Rixie Tiffany and Tan, Jia Qing and Hoque, Enamul and Joty, Shafiq , booktitle=emnlp, year=. Open

  24. [32]

    Li, Yanran and Su, Hui and Shen, Xiaoyu and Li, Wenjie and Cao, Ziqiang and Niu, Shuzi , booktitle=IJCNLP, year=. Daily

  25. [33]

    Liu, Yang and Iter, Dan and Xu, Yichong and Wang, Shuohang and Xu, Ruochen and Zhu, Chenguang , booktitle=emnlp, year=. G-eval:

  26. [34]

    On the benchmarking of

    Mendon. On the benchmarking of. Proceedings of the 6th Workshop on NLP for Conversational AI (NLP4ConvAI 2024) , year=

    2024

  27. [35]

    Contextualized sequence likelihood: Enhanced confidence scores for natural language generation , author=

  28. [36]

    Language model cascades: Token-level uncertainty and beyond , author=

  29. [37]

    Calibrating long-form generations from large language models , author=

  30. [38]

    Uncertainty estimation in autoregressive structured prediction , author=

  31. [39]

    Chain-of-thought prompting elicits reasoning in large language models , author=

  32. [40]

    arXiv preprint arXiv:1609.08144 , year=

    Google's neural machine translation system: Bridging the gap between human and machine translation , author=. arXiv preprint arXiv:1609.08144 , year=

    arXiv

  33. [41]

    arXiv preprint arXiv:2307.10236 , year=

    Look before you leap: An exploratory study of uncertainty measurement for large language models , author=. arXiv preprint arXiv:2307.10236 , year=

    arXiv

  34. [42]

    How can we know what language models know? , author=

  35. [43]

    Selective classification for deep neural networks , author=

  36. [44]

    Red teaming language models with language models , author=

  37. [45]

    arXiv preprint arXiv:2209.07858 , year=

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned , author=. arXiv preprint arXiv:2209.07858 , year=

    arXiv

  38. [46]

    Peinelt, Nicole and Nguyen, Dong and Liakata, Maria , booktitle=acl, year=

  39. [47]

    Orgad, Hadas and Toker, Michael and Gekhman, Zorik and Reichart, Roi and Szpektor, Idan and Kotek, Hadas and Belinkov, Yonatan , booktitle=iclr, year=

  40. [48]

    Factual confidence of

    Mahaut, Mat. Factual confidence of

  41. [49]

    Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation , author=

  42. [50]

    Calibrating large language models with sample consistency , author=

  43. [51]

    Do large language models know what they don’t know? , author=

  44. [52]

    Flue: Streamlined uncertainty estimation for large language models , author=

  45. [53]

    Journal of the Royal Statistical Society: Series D (The Statistician) , year=

    The comparison and evaluation of forecasters , author=. Journal of the Royal Statistical Society: Series D (The Statistician) , year=

  46. [54]

    Predicting good probabilities with supervised learning , author=

  47. [55]

    Obtaining well calibrated probabilities using

    Naeini, Mahdi Pakdaman and Cooper, Gregory and Hauskrecht, Milos , booktitle=aaai, year=. Obtaining well calibrated probabilities using

  48. [56]

    Monthly Weather Review , year=

    Verification of forecasts expressed in terms of probability , author=. Monthly Weather Review , year=

  49. [57]

    How can we know when language models know? on the calibration of language models for question answering , author=

  50. [58]

    A close look into the calibration of pre-trained language models , author=

  51. [59]

    On the calibration of large language models and alignment , author=

  52. [60]

    Dubey, Abhimanyu and Jauhri, Abhinav and Pandey, Abhinav and Kadian, Abhishek and Al-Dahle, Ahmad and Letman, Aiesha and Mathur, Akhil and Schelten, Alan and Yang, Amy and Fan, Angela and others , journal=. The

  53. [61]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    arXiv

  54. [62]

    Deepseek-r1: Incentivizing reasoning capability in

    Guo, Daya and Yang, Dejian and Zhang, Haowei and Song, Junxiao and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Zhang, Ruoyu and Ma, Shirong and Bi, Xiao and others , journal=. Deepseek-r1: Incentivizing reasoning capability in

  55. [63]

    Self-consistency improves chain of thought reasoning in language models , author=

  56. [64]

    Xiong, Miao and Hu, Zhiyuan and Lu, Xinyang and Li, Yifei and Fu, Jie and He, Junxian and Hooi, Bryan , booktitle=iclr, year=. Can

  57. [65]

    Wu, Yuhao and Hee, Ming Shan and Hu, Zhiqiang and Lee, Roy Ka-Wei , booktitle=iclr, year=

  58. [66]

    Longwriter: Unleashing 10,000+ word generation from long context

    Bai, Yushi and Zhang, Jiajie and Lv, Xin and Zheng, Linzhi and Zhu, Siqi and Hou, Lei and Dong, Yuxiao and Tang, Jie and Li, Juanzi , booktitle=iclr, year=. Longwriter: Unleashing 10,000+ word generation from long context

  59. [67]

    A survey of confidence estimation and calibration in large language models , author=

  60. [68]

    Efficient out-of-domain detection for sequence to sequence models , author=

  61. [69]

    Out-of-distribution detection and selective generation for conditional language models , author=

  62. [70]

    Inference-time intervention: Eliciting truthful answers from a language model , author=

  63. [71]

    The internal state of an

    Azaria, Amos and Mitchell, Tom , booktitle=findings-emnlp, year=. The internal state of an

  64. [72]

    A survey of uncertainty estimation methods on large language models , author=

  65. [73]

    arXiv preprint arXiv:2002.09831 , year=

    On the role of dataset quality and heterogeneity in model confidence , author=. arXiv preprint arXiv:2002.09831 , year=

    2002 arXiv

  66. [74]

    On uncertainty calibration and selective generation in probabilistic neural summarization: A benchmark study , author=

  67. [75]

    SUE: Sparsity-based Uncertainty Estimation via Sparse Dictionary Learning , author=

  68. [76]

    Selective question answering under domain shift , author=

  69. [77]

    Uncertainty estimation in large language models to support biodiversity conservation , author=

  70. [78]

    TokUR: Token-Level Uncertainty Estimation for Large Language Model Reasoning , author=

  71. [79]

    BaseCal: Unsupervised Confidence Calibration via Base Model Signals , author=

  72. [80]

    Grace: A granular benchmark for evaluating model calibration against human calibration , author=

  73. [81]

    Calibration Across Layers: Understanding Calibration Evolution in LLMs , author=

  74. [82]

    Calibrating large language models using their generations only , author=

  75. [83]

    Conftuner: Training large language models to express their confidence verbally , author=

  76. [84]

    Teaching models to express their uncertainty in words , author=

  77. [85]

    Confidence regulation neurons in language models , author=

  78. [86]

    arXiv preprint arXiv:1112.5745 , year=

    Bayesian active learning for classification and preference learning , author=. arXiv preprint arXiv:1112.5745 , year=

    arXiv

  79. [87]

    Deep think with confidence , author=

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.