{"id":"5a97da9d-098b-4e1c-ba3a-28fdfdb17d95","arxiv_id":"2506.18183","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Reasoning language models are systematically overconfident, deeper reasoning makes them more overconfident, and a two-stage introspective prompting method improves calibration for some models.","lead":"This paper tests whether advanced reasoning AI models can judge when their own answers are wrong. It finds that these models are usually overconfident, become more overconfident with deeper thinking, and that a two-stage introspective prompting technique improves confidence estimates for some, but not all, models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The introspection claim assumes the exposed 'reasoning trace' is the trace that produced the first-stage answer; for closed models the trace is elicited post hoc, so the calibration gain may be a second-pass text effect rather than introspection.","rationale":"The reader's weakest assumption is the same one I regard as most load-bearing: the introspective UQ result depends on the exposed trace being a faithful record of the first-stage reasoning. For closed models the paper cannot inspect the hidden trace, and Appendix A shows the authors had to ask the model to include its reasoning trace in the response, making the exposed trace a generated artifact rather than a verified record. The concern is not that the empirical calibration changes are dishonest or impossible; they may be real and practically useful. The concern is that the central interpretation—that reasoning about one's own chain of thought improves calibration—is underdetermined by the data. A second pass over any self-generated text that includes an instruction to 'identify flaws' could produce similar downward shifts in confidence. The paper's four-model, five-dataset evaluation is substantial, and the per-model consistency (DeepSeek and o3-Mini improve, Claude degrades) is a useful empirical observation even without the mechanistic interpretation. The reader's CONDITIONAL verdict already captures this as a condition, so my read does not change it; the concrete open-model replication above would settle whether the concern actually lands.","tokens_in":28847,"tokens_out":6447,"duration_ms":82280,"concrete_test":"Re-run the introspection experiment on DeepSeek-R1 (or another model whose hidden decoding can be logged) with three second-stage inputs: (1) the verbatim hidden trace sampled before the final answer; (2) a post-hoc explanation generated after the final answer with the trace withheld; and (3) no trace. Measure the ECE change relative to first-stage confidence on SimpleQA and GPQA. If inputs (1) and (2) produce equivalent ECE changes, trace fidelity is not load-bearing for the calibration gain; if (2) and (3) differ but (1) and (2) do not, the claim should be reworded as 'second-pass critique of generated text' rather than introspection on the actual chain of thought.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central positive result (abstract; Section 5.5) is that two-stage 'introspective UQ' improves calibration by having a second model reason about the first model's chain-of-thought trace. The mechanism requires the trace to be the reasoning that actually produced the first-stage answer. The paper cannot verify this for closed models: Appendix A states that for models without native trace access the authors append 'You must include your reasoning trace in your response,' and Section 5.5 attributes Claude's degradation to 'the implementation details of Claude, which is closed-source.' If the exposed trace is a post-hoc summary or confabulation, the experiment compares a second pass over generated text, not reasoning about the first model's internal chain of thought. The observed ECE reduction (e.g., DeepSeek and o3-Mini on SimpleQA) would still be a useful black-box finding, but it would not support the mechanism claimed in the title and abstract, and it would leave open why Claude behaves differently. This assumption is load-bearing because the method's name, motivation, and the abstract's third finding all rest on the trace being a faithful record of the reasoning that produced the answer.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies calibration of four state-of-the-art reasoning language models (Claude 3.7 Sonnet, DeepSeek R1, Gemini 2.0 Flash Thinking, o3-Mini) on five QA benchmarks (ARC-Challenge, GPQA, MMLU, StrategyQA, SimpleQA) using self-verbalized confidence estimates. It reports three main findings: (i) reasoning models are typically overconfident, especially on harder benchmarks; (ii) deeper reasoning improves accuracy but can worsen calibration, making models more confident on incorrect answers; and (iii) a proposed two-stage 'introspective UQ' procedure, where a second model reasons about the first model's reasoning trace and confidence, improves calibration for o3-Mini and DeepSeek R1 but degrades calibration for Claude 3.7 Sonnet. The paper also examines prompt-strategy invariance and the accuracy-calibration correlation.","tokens_in":29005,"tokens_out":3728,"duration_ms":41374,"significance":"If the empirical findings hold, this is a practically useful and timely study: introspective UQ would offer a no-retraining method for reducing overconfidence on hard questions, and the inference-time-scaling calibration cost would be an important consideration for system builders. The paper is commendable for using multiple models and datasets, for fixing the prompt templates as design choices rather than fitting them to the data, and for including extensive reasoning-trace examples in Appendix F that will aid replication. However, the central mechanism claim depends on the faithfulness of elicited reasoning traces, and several quantitative claims lack statistical support, so the current evidence is suggestive rather than conclusive.","major_comments":[{"comment":"The claim of a 'strong negative correlation between accuracy and ECE across all models' rests on five data points per model (one per dataset), with no error bars, repeated runs, or significance tests. For Gemini the reported r = -1 must be treated as an artifact of n = 5 rather than evidence of a perfect monotonic relationship; the same applies to the near-perfect correlations for DeepSeek and o3-Mini. Please provide bootstrap confidence intervals, per-prompt scatter plots, and a permutation test for the Spearman correlation before this correlation is used as a central finding.","section":"Section 5.3, Figure 5"},{"comment":"The reasoning-depth manipulation is not reproducible. The text states that the authors 'estimate the number of thinking tokens corresponding to these levels' for Claude and specify this as a thinking budget, but neither the estimated token counts nor the API parameters (e.g., reasoning_effort for o3-mini) are reported. Because the paper's second finding is precisely that calibration degrades with deeper reasoning after accuracy saturates, the exact operationalization of 'depth' is load-bearing; please report the token budgets for RE-Low/RE-Medium/RE-High, the model-specific mapping, and the average token counts actually consumed.","section":"Section 5.4"},{"comment":"The introspective mechanism is not established for closed models. For models without native trace access, the authors append 'You must include your reasoning trace in your response' (Appendix A), so the second stage receives a trace that may be a post-hoc summary or confabulation rather than the chain of thought that produced the first-stage answer. Under that alternative, the observed ECE decreases for DeepSeek and o3-Mini would show that a second pass over generated text changes confidence, not that explicit reasoning about one's own thought process improves calibration. This distinction is load-bearing because the method's name, motivation, and the abstract's third claim all rely on the trace being faithful. A concrete test would be, on DeepSeek, comparing introspective UQ using the API-returned native reasoning_content with introspective UQ using the appended-instruction elicited trace; if the improvements persist in both, the text-based mechanism is supported, and if not, the introspection claim needs to be weakened. The attribution of Claude's degradation to 'the implementation details of Claude, which is closed-source' (Section 5.5) should be replaced by this kind of controlled comparison.","section":"Section 5.5 and Appendix A"}],"minor_comments":[{"comment":"The number of bins M used for ECE and MCE is never specified anywhere in the paper; please state the value of M for all computations. The notation 'conf(Bm)' should also be 'conf(B_m)' for consistency with 'acc(B_m)'.","section":"Equations (3) and (5)"},{"comment":"Claims that prompt strategies do not 'significantly outperform' basic prompting are not backed by significance tests or confidence intervals; please soften the wording to 'we do not observe a consistent advantage' or add appropriate tests.","section":"Section 5.6 and Appendix E"},{"comment":"There are several typos: 'relibability' (Appendix C), 'higly' (Figure 4 caption), 'the the reasoning trace' (Section 5.5), and 'positively calibrated' (Section 7), which appears to mean 'positively correlated with accuracy'.","section":"Appendix C and Section 5.5"},{"comment":"No code or data release is mentioned; given the API-dependent nature of the experiments, please commit to releasing prompts, raw outputs, and parsing/aggregation scripts to support replication.","section":"General"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know three things about this paper. First, it is the first systematic calibration study of RL-trained reasoning models, and the main empirical findings are new: these models are overconfident, they get more overconfident as you scale reasoning effort, and a two-stage introspective prompt improves calibration for some models but not all. Second, the 'deeper reasoning hurts calibration' result is the real contribution here; it is a hidden cost of inference-time scaling that practitioners should care about. Third, the introspection result is promising but the mechanism is underdetermined for closed models because the trace handed to the second stage is elicited post hoc, not verified to be the actual reasoning that produced the answer.\n\nWhat the paper does well: the evaluation spans four frontier models and five datasets, the prompt strategies are clearly specified, and the reliability diagrams make the overconfidence pattern easy to see. The finding that prompt engineering has little effect on reasoning models, unlike non-reasoning LLMs, is a useful negative result. The paper is honest about some limitations, and the self-cited survey is background only, which is fine.\n\nSoft spots, in proportion. The statistical analysis is thin: no error bars, no repeated runs, no significance tests, and the accuracy-ECE correlation is computed over five points per model. That makes the correlation claims suggestive rather than established. The reasoning-depth comparison has an unreported mapping from OpenAI effort levels to Claude token budgets; without that mapping in the paper, the comparison is not reproducible. And the introspection mechanism assumes the exposed trace is a faithful record of the first-stage reasoning; for closed models this is unverifiable, and the paper itself notes Claude's degradation is attributed to 'closed-source implementation details.' The stress-test concern about post-hoc traces lands: the calibration gain could be a second-pass text effect rather than introspection about one's own reasoning. However, even as a black-box finding, the result is still useful for deployment, so this is a limitation to be fixed, not a fatal flaw.\n\nWho is this for: anyone building on reasoning models for high-stakes QA, and researchers working on self-verbalized UQ. It deserves a serious referee; the core questions are timely and the main claims are plausible, but the paper needs statistical rigor and a clear treatment of the trace-fidelity issue before I would rely on it. I would engage with it, and the authors should be pushed to add error bars, document the token-budget mapping, and either test on an open-weights model with verifiable traces or soften the mechanism claim.","headline":"A useful first systematic look at calibration in RL-trained reasoning models, with a genuinely new finding about inference-time scaling and a solid but not airtight introspection result; worth refereeing after fixes to the statistics and the trace-fidelity assumption.","tokens_in":29586,"tokens_out":909,"would_cite":true,"duration_ms":13023,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reasoning models are overconfident, and thinking longer makes them more so — but a critical second pass can recalibrate some of them.","keywords":["reasoning models","uncertainty quantification","calibration","expected calibration error","introspective uncertainty quantification","self-verbalized confidence","inference-time scaling","hallucination"],"falsifier":"Run introspective UQ with the reasoning trace redacted — feed the second stage only the question, the first answer, and the first confidence — and compare ECE against the full-trace condition; if calibration improves just as much without the trace, the paper's claim that reasoning about one's own chain of thought drives the gain is falsified.","tokens_in":28607,"feed_emoji":"🎯","tokens_out":5294,"duration_ms":50920,"temperature":0.7,"pith_summary":"Reasoning language models — AI systems trained to work through problems step by step — often report confidence above 85 percent even when their answers are wrong, and this paper sets out to measure how bad that miscalibration is and whether it can be fixed without retraining. It reports three findings: reasoning models are typically overconfident; giving them more time to reason makes them even more overconfident on questions they get wrong; and a two-stage \"introspective\" prompt, in which a fresh model pass reads the first pass's reasoning trace and looks for flaws, improves calibration for o3-Mini and DeepSeek R1 but degrades it for Claude 3.7 Sonnet. If confirmed, the result gives system builders a no-retraining lever for improving confidence estimates from closed models, and it shows that inference-time scaling carries a calibration cost that current benchmarks hide.","feed_headline":"Reasoning AI gets overconfident the longer it thinks","feed_subtitle":"A second, critical pass over the reasoning trace improves confidence estimates for some models, not all.","key_machinery":"The load-bearing mechanism is introspective uncertainty quantification (introspective UQ), a two-stage self-verbalized confidence procedure: stage one answers the question with a reasoning trace and a confidence score, and stage two, in a fresh session, reads that trace and returns a revised confidence in the original answer. Three introspection prompts vary in conservativeness: IUQ-Low merely asks the second pass to reason about uncertainty, IUQ-Medium asks it to identify flaws while still showing the first confidence, and IUQ-High asks for flaws with the first confidence withheld. The second stage never revises the answer, by design, to stop the model from vacillating on hard questions. Calibration is measured with expected calibration error (ECE) and maximum calibration error (MCE), bin-wise comparisons of stated confidence against observed accuracy.","core_discovery":"The paper's central claim is that the confidence statements produced by current reasoning models should not be read at face value, and that calibration defects follow a consistent pattern: overconfidence grows as reasoning effort grows, and overconfidence is concentrated on hard problems. On the remedy side, it claims that asking a second model instance to critique the first instance's chain of thought before issuing a confidence estimate can reduce expected calibration error on hard benchmarks, with more critical prompts (explicitly asking for flaws, and withholding the first confidence) working best. The same procedure is not a universal fix: one of the four models tested, Claude 3.7 Sonnet, becomes more overconfident under introspection, and the paper attributes this to closed implementation details it cannot inspect.","pith_inferences":["Inference beyond the paper: if the exposed traces are faithful, the two-stage procedure could be used to generate calibration labels for fine-tuning, since the second pass produces conservative confidence without needing ground-truth answers.","Inference beyond the paper: comparing IUQ variants suggests a testable extension where introspection is staged multiple times, with each pass seeing flaws found by the previous pass, to see whether calibration keeps improving or oscillates.","Inference beyond the paper: for closed models, the degradation seen in Claude leaves open the possibility that the sign of introspection's effect depends on how the visible trace relates to the hidden reasoning process, not on any intrinsic property of self-reflection.","Inference beyond the paper: a direct measurement of whether second-stage confidence predicts first-stage accuracy better than first-stage confidence does would isolate where the calibration gain comes from."],"forward_implications":["Confidence scores from reasoning models should be treated as upper bounds on reliability rather than calibrated probabilities, especially on hard or recent benchmarks.","Budgeting for inference-time scaling should include a calibration cost: longer thinking can increase accuracy while also pushing confidence further above accuracy on the questions the model still gets wrong.","A second, critical pass over a saved reasoning trace is a practical, finetune-free way to recalibrate closed models when the second pass is more conservative, as in IUQ-Medium and IUQ-High.","Benchmarks where models already score near ceiling are uninformative for calibration, because high accuracy masks overconfidence; new UQ-specific benchmarks are needed.","Prompt strategies that help non-reasoning models (chain-of-thought, multi-step) do not reliably change reasoning-model calibration, so prompt engineering is not a substitute for introspection or retraining."],"supporting_citations":[{"why":"Defines perfect calibration and supplies the bin-wise ECE and MCE metrics the paper uses throughout.","marker":"[17]"},{"why":"Provides the prior observation that RLHF harms token-level calibration, motivating the paper's self-verbalized UQ approach.","marker":"[37]"},{"why":"Supplies the prompt templates and confidence-elicitation strategies that the paper starts from and extends.","marker":"[53]"},{"why":"Contributes the ARC-Challenge dataset used to measure calibration on a knowledge-retrieval-style benchmark.","marker":"[8]"},{"why":"Contributes the graduate-level GPQA benchmark on which reasoning models are shown to be poorly calibrated.","marker":"[39]"},{"why":"Contributes SimpleQA, the adversarial short-form factuality benchmark where overconfidence is clearest.","marker":"[50]"},{"why":"Supplies the human analogue that longer deliberation raises confidence without changing answers, which the paper compares with deeper reasoning.","marker":"[41]"},{"why":"Introduced chain-of-thought prompting, one of the prompt strategies the paper compares against introspection.","marker":"[49]"}],"fun_headline_variants":["Longer reasoning inflates AI confidence; self-critique partly fixes it","Deep thinking makes AI overconfident; introspective checks help only some","Reasoning depth boosts AI overconfidence; only some models can self-correct","AI gets more overconfident the longer it thinks; second look helps some"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the reasoning trace handed to the second stage is a faithful record of the reasoning that actually produced the first answer and its confidence, and for closed models that trace cannot be verified.","fun_headline_variants_meta":{"raw":{"variants":["Longer reasoning inflates AI confidence; self-critique partly fixes it","Deep thinking makes AI overconfident; introspective checks help only some","Reasoning depth boosts AI overconfidence; only some models can self-correct","AI gets more overconfident the longer it thinks; second look helps some"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001554,"raw_usage":{"total_tokens":6220,"prompt_tokens":965,"completion_tokens":5255,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":5173}},"tokens_in":581,"tokens_out":5255,"duration_ms":40647,"temperature":1.0,"reasoning_tokens":5173,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:24:16.827207+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run introspective UQ with the reasoning trace redacted — feed the second stage only the question, the first answer, and the first confidence — and compare ECE against the full-trace condition; if calibration improves just as much without the trace, the paper's claim that reasoning about one's own chain of thought drives the gain is falsified.","supporting_citations":[],"review_version":1}