The reviewed record of science sign in
Pith

arxiv: 2606.11211 · v1 · pith:3GMP5TKF · submitted 2026-04-24 · cs.CL · cs.AI· cs.LG

Calibration Drift Under Reasoning: How Chain-of-Thought Budgets Induce Overconfidence in Large Language Models

Prakul Sunil Hiremath , Harshit R. Hiremath This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-07-04 19:30 UTCglm-5.2pith:3GMP5TKFrecord.jsonopen to challenge →

classification cs.CL cs.AIcs.LG
keywords reasoningcalibrationmodelmodelsunderaccuracybehaviorbudget
0
0 comments X

The pith

More reasoning can make LLMs confidently wrong

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that increasing a language model's chain-of-thought reasoning budget does not monotonically improve calibration—the relationship between how confident a model is and how often it is right. Instead, the authors identify a phenomenon they call Calibration Drift Under Reasoning (CDUR), where calibration error follows a non-monotonic trajectory: initially improving as reasoning corrects surface errors, then worsening as extended chains produce internally consistent but factually incorrect explanations that inflate confidence, and in some cases improving again at very heavy budgets when the model manages to self-correct. The central mechanism is a Hypothesis Lock-In model: in autoregressive generation, the model's early tokens function as an initial hypothesis that subsequent reasoning tokens are conditioned on and reinforce, making course correction increasingly unlikely as the chain grows. The authors evaluate Llama-3.1-8B on 47 reasoning-trap questions across four reasoning budgets and find that calibration is worst at light reasoning (ECE=0.104) and best at heavy reasoning (ECE=0.015), with a persistent overconfidence gap exceeding 0.25 at all budget levels. They propose CABStop, a stopping rule that halts reasoning when verbalized confidence diverges from an auxiliary accuracy estimate. The paper is transparent that the evidence is preliminary: 47 questions, a 42% response validity rate with possible selection bias, incomplete data for the 70B model, and large standard deviations across seeds.

Core claim

The core discovery is that calibration error ECE(B) is non-monotonic in reasoning budget B for Llama-3.1-8B on reasoning-trap questions, with worst calibration at light reasoning rather than at the extremes. The authors attribute this to a Hypothesis Lock-In mechanism: early reasoning tokens commit the model to an initial hypothesis, and subsequent tokens—being conditioned on that hypothesis—reinforce it rather than correcting it, inflating confidence without proportionally inflating correctness. The overconfidence gap (confidence minus accuracy) remains above 0.25 across all budget levels even as accuracy rises from 0.461 to 0.739, indicating a systematic confidence-accuracy decoupling.

What carries the argument

The paper introduces three central constructs. First, the reasoning budget B: an upper bound on intermediate reasoning tokens, discretized as none, light, medium, and heavy (approximately 0, 128, 512, and 2048 tokens). Second, the Commitment Model (Definition 4.1): a stylized model of autoregressive reasoning where the model implicitly samples an initial hypothesis h0 from its first few tokens, then generates subsequent reasoning tokens conditioned on h0, making the chain a constrained trajectory rather than a free search. Third, CABStop: an optimal stopping rule that monitors the gap between the model's verbalized confidence and an auxiliary accuracy estimate (e.g., from self-consistency) ,

If this is right

  • If CDUR generalizes beyond Llama-3.1-8B, inference-time compute scaling strategies that allocate more reasoning tokens to harder problems could systematically produce overconfident wrong answers at intermediate budget levels, creating a safety risk that is invisible to accuracy-only benchmarks.
  • Model evaluation protocols should report (Accuracy(B), ECE(B)) curves across multiple reasoning budgets, not just peak accuracy, because a model with acceptable accuracy at heavy reasoning may have dangerous calibration at lighter budgets where deployment is cheaper.
  • RLHF training procedures that reward fluent, confident, internally coherent responses may amplify the lock-in mechanism, suggesting that calibration-aware reward shaping—penalizing high verbalized confidence on wrong answers—could be a direct training intervention.
  • The parallel to human post-hoc rationalization suggests CDUR may be a general property of any system trained to produce justified conclusions rather than accurate ones, extending beyond LLMs to other generate-then-justify architectures.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The Commitment Model's key assumption—that reasoning tokens are conditioned on an initial hypothesis h0 and reinforce it monotonically—is partially contradicted by the paper's own data: accuracy rises from 0.461 to 0.739 across budgets, showing the model does correct errors rather than being fully locked in. A more precise model might distinguish between questions where the initial hypothesis is c
  • The CABStop algorithm's failure case where the model is consistently wrong across self-consistency samples is directly connected to the lock-in mechanism: if all k independent samples commit to the same wrong h0, self-consistency will report high agreement, CABStop will not halt, and the overconfident wrong answer will pass through. This suggests CABStop needs an accuracy signal that is orthogonal

Load-bearing premise

The load-bearing premise is that when a language model starts reasoning, its early tokens commit it to an initial hypothesis that subsequent reasoning tokens reinforce rather than revise. If real models frequently change their mind mid-chain, the lock-in mechanism breaks, confidence does not monotonically inflate, and the CDUR phenomenon may not hold.

What would settle it

If one measured ECE(B) across many models and reasoning budgets and found that calibration error is monotonically decreasing in B, or that it increases without ever decreasing, the CDUR phenomenon as defined would not hold. More specifically, if one could show that autoregressive LLMs revise their initial hypothesis mid-chain frequently enough that confidence does not inflate independently of accuracy, the central mechanistic explanation would fail.

Figures

Figures reproduced from arXiv: 2606.11211 by Harshit R. Hiremath, Prakul Sunil Hiremath.

Figure 1
Figure 1. Figure 1: Hypothesis Lock-In. The initial hypothesis [PITH_FULL_IMAGE:figures/full_fig_p027_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Schematic of the cdur phenomenon. ECE(B) is U-shaped in the reasoning budget B under the theoretical model. The green region (B < B∗ ) is the beneficial reasoning regime; the orange region (B > B∗ ) is the overconfidence regime. Empirical results for 8B are qualitatively consistent with non-monotone dynamics, but do not perfectly match this schematic (see Section 6.2) [PITH_FULL_IMAGE:figures/full_fig_p02… view at source ↗
Figure 3
Figure 3. Figure 3: Distribution of wrong-and-confident responses ( [PITH_FULL_IMAGE:figures/full_fig_p030_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: CABStop control flow. At each reasoning checkpoint, the algorithm compares the model’s expressed confidence pˆt with an auxiliary accuracy estimate αˆt . Reasoning halts when their gap exceeds the calibration threshold δ [PITH_FULL_IMAGE:figures/full_fig_p031_4.png] view at source ↗
read the original abstract

The ability of large language models (LLMs) to express calibrated uncertainty is important for safe deployment. Chain-of-thought (CoT) reasoning is widely used to improve accuracy and reliability, but its effect on calibration is not fully understood. We show that this picture is incomplete: in some settings, increasing the reasoning budget beyond a task-specific threshold can cause models to become systematically overconfident, assigning high confidence to incorrect answers. We call this phenomenon Calibration Drift Under Reasoning (CDUR) and study it both theoretically and empirically. We define reasoning budget B and analyze conditions under which Expected Calibration Error ECE(B) follows a non-monotonic pattern: it first decreases as reasoning corrects errors, then increases as longer reasoning produces internally consistent but incorrect explanations. We propose a Hypothesis Lock-In model based on autoregressive generation to explain this behavior. We evaluate Llama-3.1-8B and Llama-3.3-70B on 47 reasoning-trap questions across four reasoning budgets and three seeds (1,368 API calls; 574 valid responses). The 8B model shows non-monotonic calibration behavior, while results for the 70B model are limited to baseline evaluation and are inconclusive for budget-dependent effects. We introduce CABStop, a calibration-aware stopping rule that halts reasoning when confidence diverges from an auxiliary accuracy estimate. These results suggest that increasing reasoning depth does not always improve reliability and should be monitored carefully.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 4 minor

Summary. The manuscript introduces 'Calibration Drift Under Reasoning' (CDUR), defined as a non-monotone relationship between reasoning budget B and Expected Calibration Error ECE(B). A theoretical 'Hypothesis Lock-In' model is proposed, arguing that autoregressive generation conditioned on an initial hypothesis inflates confidence without proportionally increasing accuracy. Empirical evaluation on Llama-3.1-8B uses 47 reasoning-trap questions across four budget levels, reporting non-monotone ECE dynamics. A calibration-aware stopping rule (CABStop) is also introduced. The paper is transparent about its limitations, explicitly noting the small sample size, the 42% validity rate, and the incomplete 70B evaluation.

Significance. The paper addresses a timely and practically relevant question: whether chain-of-thought reasoning degrades uncertainty calibration. The formalization of reasoning budget effects on ECE and the proposal of a stopping rule (CABStop) are conceptually useful. The authors deserve credit for openly disclosing the 42% validity rate, the incomplete 70B evaluation, and the large standard deviations, rather than obscuring these limitations. However, the central empirical claim is not adequately supported by the current data, and the theoretical model's predictions are contradicted by the observed ECE trajectory.

major comments (3)
  1. Definition 3.3 formally requires ECE(B) to be U-shaped: dECE/dB < 0 for B < B* and dECE/dB > 0 for B > B*. The abstract and Section 3.3 state that ECE should 'first decrease... then increase.' However, Table 3 shows the opposite pattern for Llama-3.1-8B: ECE rises from 0.044 (none) to 0.104 (light), then falls to 0.050 (medium) and 0.015 (heavy). This is an inverted-U shape, directly contradicting the formal definition of CDUR. Section 6.2 retreats to the weaker claim of 'partial CDUR behavior' based on non-monotonicity, but non-monotonicity is trivially satisfied by nearly any four-point sequence and does not require the Hypothesis Lock-In mechanism. The specific mechanism predicted by Propositions 3.4-3.6 (confidence inflation under extended reasoning causing ECE to rise at high B) is not observed; ECE is lowest at heavy reasoning (0.015). The central claim is thus not supported by the
  2. The load-bearing premise of the theoretical model (Definition 4.1, Appendix A.2) is that P_theta(R_i in V_h0 | h0, R_<i) is non-decreasing in i, meaning the model locks into its initial hypothesis. If this lock-in holds, heavy reasoning should produce the worst calibration by inflating confidence on incorrect answers. The empirical data contradicts this: accuracy rises from 0.461 (none) to 0.739 (heavy), showing the model successfully corrects errors with extended reasoning. The paper acknowledges this in Section 9.4 ('Real autoregressive LLMs can revise their hypothesis mid-chain'), but this admission breaks the core mechanism. If models can revise mid-chain, the lock-in model does not hold, and the propositions do not apply to the observed data.
  3. The empirical evidence is too weak to support the claims. Table 3 reports ECE=0.050±0.049 at medium budget, a standard deviation nearly equal to the mean. With only 47 questions, 3 seeds, and a 42% validity rate (574/1368 responses), the effective per-budget sample size is approximately 40-50 valid responses. The paper itself acknowledges in Section 6.6 that the medium-budget ECE (0.050) and no-reasoning ECE (0.044) are 'not statistically distinguishable.' Given these limitations, the non-monotone trajectory may be noise rather than a robust phenomenon.
minor comments (4)
  1. Section 5.3: The 42% validity rate raises a selection bias concern. If invalid responses correlate with uncertainty, the valid subset oversamples confident responses, potentially inflating the overconfidence gap (OG). The paper discusses this but does not test it. A sensitivity analysis on the format-compliant subset would strengthen the claims.
  2. Table 2: The 'smoking gun' examples are all from the no-reasoning condition. If the lock-in mechanism is the primary concern, showing wrong-and-confident examples that persist or worsen from light to heavy reasoning would be more illustrative of the proposed mechanism.
  3. Figure 2: The schematic shows a U-shaped ECE curve, but the empirical data (Table 3) shows an inverted-U. The figure should be updated to reflect the actual observed trajectory, or the discrepancy should be more prominently flagged in the caption.
  4. Section 7.2, Proposition 6.1: The optimality of CABStop relies on the assumption that confidence p_t is non-decreasing. The paper acknowledges in Section 7.5 that verbalized confidence may fluctuate. Since the stopping rule's optimality guarantee requires an assumption the paper itself admits is often violated, the practical utility of CABStop is unclear without empirical validation.

Simulated Author's Rebuttal

3 responses · 1 unresolved

We thank the referee for a careful and substantive reading of our manuscript. The referee raises three major points, all of which we take seriously. We agree that the current manuscript overclaims relative to what the data supports, and we propose concrete revisions to address each point. Below we respond to each major comment in turn.

read point-by-point responses

  1. Referee: Definition 3.3 formally requires ECE(B) to be U-shaped (first decreasing, then increasing), but Table 3 shows the opposite pattern for 8B: ECE rises from 0.044 (none) to 0.104 (light), then falls to 0.050 (medium) and 0.015 (heavy). This is an inverted-U, contradicting the formal definition. Section 6.2 retreats to 'partial CDUR' based on non-monotonicity, which is trivially satisfied by any four-point sequence and does not require the Hypothesis Lock-In mechanism.

    Authors: The referee is correct. The observed ECE trajectory (0.044 → 0.104 → 0.050 → 0.015) is an inverted-U, not the U-shape required by Definition 3.3. Our current framing—introducing a formal U-shaped definition and then retreating to 'partial CDUR' based on non-monotonicity—is internally inconsistent and weakens the paper. We will revise the manuscript to address this in the following ways: (1) We will reframe the central empirical claim as 'intermediate-reasoning overconfidence': the worst calibration occurs at light reasoning, not at the extremes. This is a genuine and practically relevant finding even though it does not match the U-shaped CDUR definition. (2) We will revise Definition 3.3 to define CDUR as the existence of a non-monotone ECE(B) with an interior maximum (inverted-U), which is what our data actually shows. The theoretical model in Section 4 would then need to be re-examined: the Hypothesis Lock-In mechanism predicts confidence inflation under extended reasoning, which should produce a U-shape (rising ECE at high B), but our data shows the opposite. We will be explicit that the theoretical model and the empirical data are in tension on this point, and we will not claim that the data confirms the model's predictions. (3) We will remove or substantially soften the claim that the data is 'consistent with CDUR' and instead state plainly that the data shows a different non-monotone pattern than the one our theory predicts. We agree that 'non-monotonicity' alone is too weak a criterion to be meaningful with four data points, and we will not rely on it as the primary claim. revision: yes

  2. Referee: The load-bearing premise of the theoretical model (Definition 4.1, Appendix A.2) is that the model locks into its initial hypothesis, meaning heavy reasoning should produce the worst calibration by inflating confidence on incorrect answers. But accuracy rises from 0.461 (none) to 0.739 (heavy), showing the model successfully corrects errors with extended reasoning. The paper acknowledges this in Section 9.4 ('Real autoregressive LLMs can revise their hypothesis mid-chain'), but this admission breaks the core mechanism.

    Authors: The referee is correct that this is a genuine tension in the paper. If the model can revise its hypothesis mid-chain—and the accuracy data clearly shows it can—then the commitment model's core assumption (non-decreasing lock-in) does not hold for the observed data, and Propositions 3.4–3.6 do not directly apply. We acknowledge this without qualification. In the revision, we will take one of two approaches, and we lean toward the first: (a) Reframe the theoretical model as applying to a specific regime—light-to-medium reasoning where partial chains reinforce an initial hypothesis without sufficient depth for self-correction. The lock-in mechanism may hold in this regime even if it breaks at heavy reasoning. The light-budget ECE of 0.104 (the worst) is consistent with this: brief reasoning locks in a wrong hypothesis and inflates confidence, while heavy reasoning escapes lock-in. We would then explicitly state that the propositions characterize only the intermediate regime, not the full budget range. (b) Alternatively, we could remove the formal propositions entirely and present the Hypothesis Lock-In model as a qualitative hypothesis that explains the intermediate-budget overconfidence peak, without claiming formal predictive power. We are inclined toward (a) because the lock-in mechanism does have qualitative support at the light budget (e.g., the syllogism error persisting from none to light), but we will be explicit that the model does not account for the heavy-budget regime where accuracy improves substantially. Either way, we will not claim that the theoretical model's predictions are confirmed by the data. revision: yes

  3. Referee: The empirical evidence is too weak: ECE=0.050±0.049 at medium budget (std nearly equal to mean), only 47 questions, 3 seeds, 42% validity rate, effective per-budget sample size ~40-50. The paper itself acknowledges medium and none ECE are not statistically distinguishable. The non-monotone trajectory may be noise.

    Authors: We agree. The referee's assessment of the statistical weakness is accurate, and we have already acknowledged most of these limitations in the manuscript (Sections 5.3, 6.6, 8). However, the referee is right that acknowledging limitations in a separate section does not fix the problem that the central empirical claim is not adequately supported. We will make the following changes: (1) We will downgrade the central empirical claim from 'the 8B model exhibits non-monotonic calibration dynamics' to 'the 8B model shows a directional pattern in which light-reasoning ECE exceeds both no-reasoning and heavy-reasoning ECE across all three seeds, though the difference is not statistically significant given the sample size.' (2) We will add a proper statistical test (e.g., a paired bootstrap or permutation test across seeds) and report p-values or confidence intervals for the key comparisons, rather than relying on point estimates with large standard deviations. (3) We will state in the abstract and conclusion that the empirical results are preliminary and require replication on a larger dataset (we note this in Section 10 but not in the abstract). (4) We cannot, in the current revision, increase the sample size or complete the 70B evaluation—these require additional compute resources we do not currently have. We will be transparent about this. We agree that if the referee considers the empirical evidence insufficient to support any empirical claim even as preliminary, the appropriate revision would be to reframe the paper as primarily a theoretical contribution with illustrative (not confirmatory) empirical observations. revision: partial

standing simulated objections not resolved
  • We cannot provide additional empirical data (larger sample size, completed 70B evaluation, or additional model families) within the revision timeframe due to compute resource constraints. The referee's concern that the non-monotone trajectory may be noise rather than a robust phenomenon is a valid empirical objection that we can only address by softening our claims, not by providing stronger data.

Circularity Check

0 steps flagged

No significant circularity; propositions are simple consequences of stated assumptions, not definitions in disguise

full rationale

The paper's theoretical chain runs: Definition 4.1 (Commitment Model) → Appendix A.2 assumption (P[R_i ∈ V_{h0} | h0, R_{<i}] non-decreasing in i) → Propositions 3.4–3.6 → Proposition 6.1 (CABStop optimality). The propositions are nearly tautological given the assumptions — e.g., Proposition 3.4 assumes consistency is non-decreasing and confidence is a non-decreasing function of consistency, then concludes confidence is non-decreasing — but this is a straightforward (if trivial) logical derivation, not circularity. The assumptions are about token-level probabilities; the conclusions are about aggregate confidence and calibration error. They are related but not identical, so the assumptions are not defined in terms of the conclusions. There are no self-citations: all references are to external work (Wei et al., Guo et al., Xiong et al., etc.). The empirical results (ECE, accuracy, OG across budgets) are independent measurements not fitted to the theoretical model. The notable weakness is that the empirical ECE trajectory (inverted-U: 0.044→0.104→0.050→0.015) contradicts the U-shape required by Definition 3.3, but this is a correctness/validity concern, not circularity. The paper is transparent about this mismatch (Section 6.2) and retreats to 'partial CDUR.' Score of 1 reflects that the propositions add little beyond restating the assumptions, but no step reduces to its inputs by definition or self-citation.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 3 invented entities

The paper introduces three invented entities: CDUR (the phenomenon), the Hypothesis Lock-In Model (the mechanism), and CABStop (the intervention). CDUR has partial empirical support. The Lock-In Model's key assumption (non-decreasing consistency reinforcement) is ad hoc and its prediction of accuracy plateau is contradicted by the data. CABStop is unvalidated.

free parameters (3)
  • Reasoning budget levels (none, light, medium, heavy) = 0, 128, 512, 2048 tokens
    Chosen by hand to discretize the budget axis; not derived from theory.
  • CABStop threshold delta = 0.10
    Recommended starting value chosen to align with conventional ECE thresholds (Section 7.5).
  • Temperature = 0.7
    Set to allow variability across seeds (Section 5.1).
axioms (4)
  • ad hoc to paper P_theta(R_i in V_h0 | h0, R_<i) is non-decreasing in i
    Appendix A.2. This is the core assumption of the Commitment Model. The paper states it holds approximately for instruction-tuned LLMs but admits it may be violated when models encounter contradictory evidence mid-chain.
  • ad hoc to paper Verbalized confidence is a non-decreasing function g of the consistency score C
    Proposition 3.4 proof sketch. Assumes confidence tracks internal consistency, which is the mechanism being tested.
  • domain assumption Initial hypothesis h0 is sampled from P_theta(.|q) and fixed during reasoning
    Definition 4.1. Idealization of autoregressive generation; Section 4.2 acknowledges h0 is not sampled explicitly in real LLMs.
  • standard math Standard ECE definition with equal-width binning
    Section 5.6. Standard calibration metric from Guo et al. 2017.
invented entities (3)
  • CDUR (Calibration Drift Under Reasoning) independent evidence
    purpose: Names the phenomenon of non-monotonic ECE(B)
    Empirically tested on Llama-3.1-8B, though evidence is limited and does not perfectly match the theoretical U-shape.
  • Hypothesis Lock-In Model no independent evidence
    purpose: Mechanistic explanation for why confidence inflates under extended reasoning
    The commitment model is an idealization. Its key prediction (accuracy plateau with confidence growth) is only partially supported: accuracy rises from 0.461 to 0.739, contradicting the plateau prediction.
  • CABStop algorithm no independent evidence
    purpose: Calibration-aware stopping rule for inference-time reasoning
    Proposed but not empirically validated in the paper. Section 7.5 states 'Empirical validation of the algorithm on a held-out question set is a necessary step before deployment.'

pith-pipeline@v1.1.0-glm · 20517 in / 2757 out tokens · 224838 ms · 2026-07-04T19:30:42.175651+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

18 extracted references · 18 canonical work pages · 6 internal anchors

  1. [1]

    S., and Robbins, H

    Chow, Y. S., and Robbins, H. (1961). On optimal stopping rules. Zeitschrift f\"ur Wahrscheinlichkeitstheorie und verwandte Gebiete, 2(1):33--49

    work page 1961

  2. [2]

    De Groot, M. H. and Fienberg, S. E. (1983). The comparison and evaluation of forecasters. Journal of the Royal Statistical Society: Series D (The Statistician), 32(1-2):12--22

    work page 1983

  3. [3]

    and Durrett, G

    Desai, S. and Durrett, G. (2020). Calibration of pre-trained transformers. Proceedings of EMNLP

    work page 2020

  4. [4]

    Graves, A. (2016). Adaptive computation time for recurrent neural networks. arXiv preprint arXiv:1603.08983

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  5. [5]

    Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. (2017). On calibration of modern neural networks. Proceedings of ICML

    work page 2017

  6. [6]

    Haidt, J. (2001). The emotional dog and its rational tail: A social intuitionist approach to moral judgment. Psychological Review, 108(4):814

    work page 2001

  7. [7]

    Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, T., et al. (2022). Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  8. [8]

    S., Reid, M., Matsuo, Y., and Iwasawa, Y

    Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. (2022). Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems, 35

    work page 2022

  9. [9]

    Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. (2023). Let's verify step by step. arXiv preprint arXiv:2305.20050

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  10. [11]

    Murphy, A. H. (1977). The value of climatological, categorical and probabilistic forecasts in the cost-loss ratio situation. Monthly Weather Review, 105(7):803--816

    work page 1977

  11. [12]

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35

    work page 2022

  12. [13]

    Snell, C., Lee, J., Xu, K., and Kumar, A. (2024). Scaling LLM test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  13. [14]

    Turpin, M., Michael, J., Perez, E., and Bowman, S. R. (2023). Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting. arXiv preprint arXiv:2305.04388

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  14. [15]

    Wald, A. (1947). Sequential Analysis. Wiley, New York

    work page 1947

  15. [16]

    Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. (2022). Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  16. [17]

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35

    work page 2022

  17. [18]

    Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., and Hooi, B. (2024). Can LLMs express their uncertainty? An empirical evaluation of confidence elicitation in LLMs. Proceedings of ICLR

    work page 2024

  18. [19]

    Zhou, W., Sha, Z., Zhang, Q., Gong, W., Shan, B., Yang, L., He, X., and Liu, B. (2023). Navigating the grey area: How expressions of uncertainty and overconfidence affect language models. arXiv preprint arXiv:2302.13439

    work page arXiv 2023