Pith. sign in

REVIEW 3 major objections 6 minor 108 references

This paper shows that the post-training paradigm (SFT, RL, or on-policy distillation) determines which stage of a reasoning trace a model's confidence can be trusted, and that a position-aware strategy improves early stopping and answer agg

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 03:50 UTC pith:75O3ODZL

load-bearing objection A genuine three-stage lens on how post-training shapes confidence in reasoning models, with a striking position-dependence result; but the PosConf gains are partly tuned on the test benchmark and Eq. (1) has a sign error. the 3 major comments →

arxiv 2607.13753 v2 pith:75O3ODZL submitted 2026-07-15 cs.CL

Post-Training Shifts Confidence: A Three-Stage Analysis of How SFT, RL, and OPD Shape CoT Calibration

Shuhao Li , Guodong Du , Anhao Zhao , Wanyu Lin , Tianyu Yuan , Xiaoyu Shen This is my paper
classification cs.CL
keywords confidence calibrationchain-of-thought reasoningpost-trainingsupervised fine-tuningreinforcement learningon-policy distillationearly stoppinganswer aggregation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that the training recipe a reasoning model receives determines not whether its confidence is reliable, but where in the reasoning process that confidence can be trusted. Using a three-stage framework—before, during, and after chain-of-thought generation—the authors show that on-policy distillation yields the best pre-reasoning difficulty signal, supervised fine-tuning yields the best online early-stopping signal, and reinforcement learning yields the best post-reasoning aggregation signal. They also show confidence reliability moves within a trace: RL confidence only becomes informative after the model commits to a solution path, while OPD confidence is useful early but can invert later. From this they build PosConf, a position-aware confidence score that reads confidence only inside reliable relative-position intervals, and report gains of 6.1 points over majority voting for RL aggregation and up to 4.3 points for OPD early stopping under tight token budgets. A sympathetic reader cares because it offers a practical recipe for spending test-time compute based on internal confidence instead of treating confidence as one fixed number.

Core claim

The paper's central discovery is that confidence derived from next-token probabilities is a stage-dependent and position-dependent signal whose reliability is shaped by the post-training objective. OPD models carry useful difficulty information before a single reasoning token is written, SFT models keep a stable confidence gap between correct and incorrect traces throughout generation, and RL models only become discriminative after a path-commitment phase—yet at that later point their trace-level confidence ranks candidates well enough to beat majority voting. The inverse side is equally load-bearing: OPD confidence, strong early, can become inversely calibrated in the tail, so naive confide

What carries the argument

The central object is a token-level confidence score c_l = -(1/k) sum over the top-k next-token probabilities log p(v | prefix), smoothed by a sliding window for online use. The paper reads this score at three decision points—Pre-CoT (query-level difficulty ranking), Intra-CoT (windowed early stopping), and Post-CoT (trace-level aggregation)—and introduces PosConf, a weighted confidence C_w(tau) that only uses token positions within a model-specific relative-position interval [alpha, beta] in normalized trace length. That interval is the mechanism that converts the position-dependence observation into a decision rule; it is derived from trajectory analysis on one benchmark and then applied a

Load-bearing premise

The claim rests on treating one training run per paradigm as a fair representative of that paradigm—so OPD's late inverse calibration could be an artifact of its distillation weight or logit scaling rather than a property of on-policy distillation itself—and on trusting patterns diagnosed on a 30-problem AIME set.

What would settle it

Sweep the OPD distillation coefficient (e.g., lambda from 0.1 to 2.0) and the RL KL/entropy coefficients, and re-plot the correct/incorrect confidence trajectories: if the late-trace crossing disappears or moves substantially, the position-dependence story is a hyperparameter artifact. Separately, recompute PosConf intervals from a different benchmark or a larger sample; if the fixed AIME-derived intervals stop improving aggregation or early stopping, the claimed transferability fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Calibration is not a single model property: a model can be reliable before reasoning and unreliable after, or vice versa, so evaluations should report confidence utility stage by stage.
  • Naive confidence filtering of complete traces helps RL models but actively hurts SFT and OPD models; aggregation strategies need to match the post-training objective.
  • Early stopping works best when the stopping rule is active only in the position interval where confidence is calibrated—PosConf's OPD gain of up to 4.3 points comes from avoiding its late inverse-calibration region.
  • PosConf raises RL confidence-filtered voting to 6.1 points above majority voting on average across the four benchmarks, using intervals fixed from one trajectory analysis rather than re-tuned per benchmark.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the objective-dependent pattern generalizes beyond maths and the 7B scale, then test-time compute policies (when to stop, how many samples to draw, which trace to trust) could be designed by reading the post-training objective rather than by tuning thresholds for every model.
  • The OPD inverse-calibration result is the most fragile-looking piece: an editorial reading is that it may depend on the distillation loss weighting. A direct test is to sweep the OPD lambda coefficient and watch whether the confidence trajectories still cross; if they stop crossing, the paradigm-level conclusion narrows to a hyperparameter-dependent statement.
  • PosConf's intervals are hand-set per model from one small diagnostic set; a natural extension is to learn the interval boundaries as part of a decision rule on each benchmark, which could recover or exceed the reported gains and reveal whether a single universal interval ordering holds across scales.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a three-stage framework for evaluating confidence in reasoning LLMs: Pre-CoT (difficulty estimation), Intra-CoT (early stopping), and Post-CoT (answer aggregation). It compares Qwen2.5-7B-Instruct variants trained with SFT, RL, and OPD on the same data mixture, and reports that OPD gives the most reliable Pre-CoT signal, SFT the strongest Intra-CoT signal, and RL the best Post-CoT signal. It further observes that confidence reliability is position-dependent, and introduces PosConf, a position-aware confidence extraction method using model-specific relative-position intervals. Experiments on AIME 2024, AIME 2025, AMC 2023, and MATH500 show gains for RL aggregation (6.1 points over majority voting) and for OPD/RL early stopping under low token budgets.

Significance. If the findings hold, the paper makes a useful contribution by showing that post-training objectives shape not just final accuracy but also the stage- and position-dependent reliability of internal confidence signals, with practical implications for test-time compute allocation. The three-stage decomposition is intuitive and the controlled backbone/data setup is a strength, as is the public code release. The results are also falsifiable and clearly presented. However, the strongest claims—particularly the paradigm-level attribution and the PosConf gains—are currently supported by a single run per paradigm and by intervals selected on the same AIME 2024 data that appear in the headline aggregation result. These issues are fixable but require additional experiments or careful reframing.

major comments (3)
  1. [Section 2, Eq. (1)] The definition c_ℓ = −(1/k) Σ_{v∈V_k} log p(v|s_ℓ) is inconsistent with the prose. For a peaked distribution, each p(v) is close to 1, so −log p(v) is close to 0 and c_ℓ is small; for a flat distribution, −log p(v) is large and c_ℓ is large. The text says the opposite: 'a highly peaked distribution results in a larger confidence value.' All subsequent uses treat higher c as higher confidence, and the trajectory plots and AUROC/PRR results only make sense under the prose convention. If a reimplementer uses the printed formula, the confidence rankings will be inverted. This needs to be corrected, e.g., by defining c_ℓ as the average top-k probability or by explicitly labeling Eq. (1) as an uncertainty measure and reversing downstream comparisons.
  2. [Section 5, Table 3] The PosConf intervals are derived from AIME 2024 trajectory analysis, but AIME 2024 is then included in the headline evaluation. The abstract's 6.1-point aggregation gain corresponds to Qwen-RL PosConf-Top-5% (63.5%) versus majority vote (57.4%) in Table 2, an average that includes AIME 2024. Similarly, Figure 2 shows PosConf curves on AIME 2024. This makes the PosConf benefit on AIME 2024 in-sample and potentially overfit to a 30-problem dataset. The paper should report out-of-sample results with intervals fixed from a separate development set, or use cross-validation, and should also provide a sensitivity analysis of the interval endpoints. Without this, the PosConf claim is not yet fully supported.
  3. [Section 3 and Appendix B] The central attribution claim—'Holding the backbone and data source fixed makes calibration differences more attributable to the post-training objective'—is not fully supported by the evidence. Each paradigm is a single run with paradigm-specific hyperparameters: SFT uses rejection-filtered teacher traces; RL uses KL coefficient 1e-4 and entropy coefficient 1e-3; OPD uses GOLD-style dense KL with λ=1.0 and universal logit distillation. The observed OPD inverse calibration later in the trace could be an artifact of the distillation weight or logit scaling rather than a general property of on-policy distillation. Similarly, RL's entropy penalty can significantly affect token distributions and could shift the position where confidence becomes discriminative. The paper should either provide multiple seeds and ablations over the key hyperparameters (at minimum OPD λ and RL entropy/KL) or soft
minor comments (6)
  1. [Appendix C] The description of 'weighted majority voting' over retained answers in confidence-filtered aggregation is vague. Please specify the weights (e.g., frequency of the answer among retained traces, or confidence-weighted).
  2. [Section 2] The quantity in Eq. (1) is called 'confidence' but it behaves like an uncertainty or entropy measure. Consider adopting a clearer terminology, such as 'predictive concentration' or 'confidence score' with an explicit definition of the monotonic direction.
  3. [Figure 2] The legend lists 'Qwen-OPD+PosConf' and 'Qwen-RL+PosConf' but not the corresponding standard-curve colors for each model; the figure is difficult to parse in grayscale. Adding markers or distinguishing line styles would help.
  4. [Section 5] The interval endpoints in Table 3 are reported without justification or error analysis. Since the underlying trajectory analysis is based on only 30 AIME 2024 problems, a quantitative comparison of alternative endpoint choices would strengthen the claim that the specific intervals are not arbitrarily chosen.
  5. [Section 4, Table 2] The Qwen-OPD PosConf-Most row shows 28.6 on AIME 2025, a non-integer value, while other entries are integers. If this is an average over repetitions, it would be helpful to report standard deviations or clearly mark that values are means.
  6. [Limitations] The limitations paragraph is honest and appropriately scoped. I encourage the authors to also mention the single-run-per-paradigm limitation and the AIME 2024 interval-selection issue there, or better, to address them directly in the main text.

Circularity Check

1 steps flagged

PosConf intervals are fit on AIME 2024 and then evaluated on AIME 2024, making part of the reported gain self-referential; the rest of the analysis is independent.

specific steps
  1. fitted input called prediction [Section 5 (Analysis), Table 3 and Table 2]
    "Table 3 summarizes the PosConf parameters derived from AIME 2024 trajectory analysis."

    The PosConf reliable intervals (e.g., Qwen-OPD Intra/Post [0,0.6], Qwen-RL Intra [0.2,1]) are read off the AIME 2024 correct-vs-incorrect confidence trajectories, and the same section then reports PosConf aggregation accuracy in Table 2 and Intra-CoT improvements in Figure 2 on AIME 2024. For AIME 2024, PosConf is not an independent prediction: the interval choice is fit to the same benchmark's trajectory separations, so part of the claimed gain on that benchmark is selected rather than predicted. The abstract's 6.1-point average gain is, however, dominated by AIME 2025/AMC/MATH500 where the intervals are a genuine transfer, so the circularity is partial.

full rationale

The central three-stage calibration results (Pre-CoT AUROC/PRR, Intra-CoT matched-budget frontiers, Post-CoT aggregation) are measured on benchmark data using standard definitions (Eqs. 1-7) and do not reduce to the paper's own inputs. No load-bearing self-citation or imported uniqueness theorem appears; citations to the authors' prior OPD work are not used to justify the empirical patterns. The main circularity concern is PosConf: Table 3's intervals are 'derived from AIME 2024 trajectory analysis,' and the paper then reports PosConf benefits on AIME 2024 (Table 2, Figure 2). This makes the AIME 2024 portion of PosConf evidence partly self-referential (interval selection and evaluation on the same 30-problem set). However, the headline 6.1-point RL aggregation gain and the low-budget OPD gains (e.g., 4.32 points on AIME 2025 at 40% budget) come from benchmarks not used to set the intervals, so the position-dependence claim has independent out-of-sample support. The 'controlled comparison' is a single run per paradigm with paradigm-specific hyperparameters; that is a robustness/attribution limitation, not an equation-level circularity, and I do not factor it into the score. Overall: partial circularity confined to the in-sample AIME 2024 PosConf evaluation, with the central stage-wise findings intact.

Axiom & Free-Parameter Ledger

7 free parameters · 6 axioms · 0 invented entities

What the paper needs from upstream that it does not establish itself: (1) PosConf interval endpoints fitted to the same AIME 2024 benchmark that appears in the headline gains; (2) the comparability assumption that one training run per paradigm supports claims about the paradigm; (3) the operational definition of confidence whose printed sign contradicts the results; (4) hand-chosen protocol constants (k=20, D=10, 5%/10% filters, temperatures) that are not swept or ablated. No invented entities; PosConf is a weighting rule, not an entity.

free parameters (7)
  • PosConf relative-position interval endpoints (α,β) per model and stage = SFT Intra [0,1]/Post [0.6,1]; RL Intra [0.2,1]/Post [0.6,1]; OPD Intra [0,0.6]/Post [0,0.6]
    Chosen by inspecting AIME 2024 correct-vs-incorrect confidence trajectories (Section 5, Figure 3, Table 3) and then evaluated on the same benchmark; the headline 6.1-pt aggregation and 4.3-pt early-stopping gains depend on these endpoints.
  • Top-k tokens k for token confidence = 20
    Hand-chosen in Appendix C ('We use top-k=20 next-token log-probabilities for all confidence computations'); shapes the smoothing behavior in Eqs. (1)-(3).
  • Sliding-window adaptive divisor D = 10
    Sets the Intra-CoT window size w_x = max{1, round(L_x/D)}; hand-chosen (Appendix C).
  • Confidence-filter retention fractions = 5% and 10%
    Top-η% filtering thresholds in Post-CoT aggregation (Table 2); hand-chosen, no sweep reported.
  • Intra-CoT sweep protocol sizes = M=50 thresholds, W=32 warmup, N=256 online
    Protocol constants (Algorithm 1); chosen without reported sensitivity analysis.
  • Sampling temperature / top-p = 0.6 / 0.95
    Generation config (Appendix C) affecting trace diversity and thus the difficulty and aggregation baselines.
  • Training-time confidence-shaping hyperparameters = RL KL 1e-4, entropy 1e-3; OPD λ=1.0
    These coefficients directly shape how peaked the models' token distributions are; the studied 'confidence signatures' are measured on models produced by these specific settings, so they are parameters of the claim, not just training logistics.
axioms (6)
  • domain assumption Top-k average negative log-probability is a valid confidence signal whose direction is 'peaked = confident'
    Eq. (1) with the prose in Section 2; the sign in the printed formula contradicts the prose, but the rank-based metrics and the early-stopping results require the prose convention.
  • domain assumption Relative position ρ=t/m is comparable across traces of different lengths
    Section 2 defines ρ_t = t/m; used to pool trajectories and define PosConf intervals; presupposes length-normalized alignment is meaningful for token-level confidence.
  • domain assumption Majority vote over sampled traces is a valid difficulty label for Pre-CoT evaluation
    Section 2 / Appendix C: 'we label a problem as solved if the majority-voted answer over sampled traces matches the ground truth'; the Pre-CoT AUROC measures ranking against this label, not against an external difficulty rating.
  • domain assumption The three training runs are comparable, so confidence differences are attributable to the paradigm
    Section 3: 'Holding the backbone and data source fixed makes calibration differences more attributable to the post-training objective'; single run per paradigm with differing supervision signals.
  • domain assumption Final-answer extraction (math equivalence, else exact match) correctly labels trace correctness
    Appendix C: 'Invalid, missing, or unparseable answers are counted as incorrect'; these labels underpin all correct-vs-incorrect analyses in Figures 3-4 and Tables 1-2.
  • domain assumption AIME 2024 trajectory patterns are representative enough to fix intervals for other benchmarks
    Section 5 / Table 3: intervals derived from AIME 2024 analysis, then applied to AIME 2025, AMC 2023, and MATH500 without per-benchmark validation shown.

pith-pipeline@v1.3.0-alltime-deepseek · 17298 in / 27853 out tokens · 244927 ms · 2026-08-02T03:50:06.306562+00:00 · methodology

0 comments
read the original abstract

Large language models have made strong reasoning gains through supervised fine-tuning, reinforcement learning, and on-policy distillation, yet these post-training methods are usually evaluated only by final-answer accuracy. We study how they reshape confidence during reasoning. We introduce a three-stage calibration framework that evaluates confidence before, during, and after chain-of-thought generation, corresponding to difficulty estimation, early termination, and answer aggregation. Through a controlled comparison on mathematical reasoning benchmarks, we find that OPD provides the most useful pre-reasoning confidence, SFT gives the strongest online signal for early stopping, and RL produces the most reliable trace-level signal for aggregation. We further show that confidence reliability is position-dependent: RL confidence becomes informative after a path-commitment phase, while OPD confidence is useful early but can become inversely calibrated later. Based on this observation, we propose PosConf, a position-aware confidence strategy that uses confidence only from reliable relative-position intervals. PosConf improves RL answer aggregation by 6.1 points over majority voting and consistently improves OPD early stopping under tight token budgets, with gains up to 4.3 points by avoiding its later inverse-calibration region, showing that \emph{confidence in reasoning models should be used both stage-wise and position-awarely}. Our code is available at https://github.com/EIT-NLP/Post-Training-Calibration.

Figures

Figures reproduced from arXiv: 2607.13753 by Anhao Zhao, Guodong Du, Shuhao Li, Tianyu Yuan, Wanyu Lin, Xiaoyu Shen.

Figure 1
Figure 1. Figure 1: Overview of the three-stage reasoning calibration framework. Confidence is evaluated before, during, and [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Matched-budget Intra-CoT early-stopping frontiers under standard confidence and PosConf. The x-axis [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Mean token-level confidence trajectories for correct and incorrect traces on AIME 2024. Qwen-SFT [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Trace-level confidence distributions for correct and incorrect traces on AIME 2024. Qwen-RL shows the [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Supplementary confidence diagnostics on AIME 2025, AMC 2023, and MATH500. These plots test [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

108 extracted references · 35 linked inside Pith

  1. [1]

    2023 , eprint=

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models , author=. 2023 , eprint=

    2023

  2. [2]

    2026 , eprint=

    Query-Level Uncertainty in Large Language Models , author=. 2026 , eprint=

    2026

  3. [3]

    2026 , eprint=

    PowerOPD: Stabilizing On-Policy Distillation with Bounded Power Transformation , author=. 2026 , eprint=

    2026

  4. [5]

    2026 , eprint=

    TAPS: Target-Aware Prefix Tree Selection for Diffusion-Drafted Speculative Decoding , author=. 2026 , eprint=

    2026

  5. [7]

    Advances in neural information processing systems , volume=

    Large language models are zero-shot reasoners , author=. Advances in neural information processing systems , volume=

  6. [8]

    2021 , eprint=

    Show Your Work: Scratchpads for Intermediate Computation with Language Models , author=. 2021 , eprint=

    2021

  7. [9]

    2022 , eprint=

    Solving Quantitative Reasoning Problems with Language Models , author=. 2022 , eprint=

    2022

  8. [10]

    Advances in neural information processing systems , volume=

    Solving quantitative reasoning problems with language models , author=. Advances in neural information processing systems , volume=

  9. [11]

    Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    Unveiling the key factors for distilling chain-of-thought reasoning , author=. Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    2025

  10. [12]

    Advances in Neural Information Processing Systems , volume=

    Star: Bootstrapping reasoning with reasoning , author=. Advances in Neural Information Processing Systems , volume=

  11. [13]

    Advances in neural information processing systems , volume=

    Tree of thoughts: Deliberate problem solving with large language models , author=. Advances in neural information processing systems , volume=

  12. [14]

    International conference on machine learning , pages=

    On calibration of modern neural networks , author=. International conference on machine learning , pages=. 2017 , organization=

    2017

  13. [15]

    Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    A survey of confidence estimation and calibration in large language models , author=. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    2024

  14. [16]

    Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V

    Uncertainty quantification and confidence calibration in large language models: A survey , author=. Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2 , pages=

  15. [18]

    2022 , eprint=

    Language Models (Mostly) Know What They Know , author=. 2022 , eprint=

    2022

  16. [20]

    International Conference on Learning Representations , volume=

    Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms , author=. International Conference on Learning Representations , volume=

  17. [21]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    2023

  18. [23]

    and Hauskrecht, Milos , title =

    Naeini, Mahdi Pakdaman and Cooper, Gregory F. and Hauskrecht, Milos , title =. Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence , pages =. 2015 , isbn =

    2015

  19. [24]

    Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

    Calibration of pre-trained transformers , author=. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

    2020

  20. [25]

    Transactions of the Association for Computational Linguistics , volume=

    How can we know when language models know? on the calibration of language models for question answering , author=. Transactions of the Association for Computational Linguistics , volume=. 2021 , publisher=

    2021

  21. [26]

    Advances in neural information processing systems , volume=

    Revisiting the calibration of modern neural networks , author=. Advances in neural information processing systems , volume=

  22. [27]

    Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods , volume =

    Platt, John , year =. Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods , volume =

  23. [28]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    Calibration Across Layers: Understanding Calibration Evolution in LLMs , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  24. [29]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    The llm already knows: Estimating llm-perceived question difficulty via hidden representations , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  25. [31]

    Advances in neural information processing systems , volume=

    Selective classification for deep neural networks , author=. Advances in neural information processing systems , volume=

  26. [33]

    Advances in neural information processing systems , volume=

    Training language models to follow instructions with human feedback , author=. Advances in neural information processing systems , volume=

  27. [34]

    Advances in neural information processing systems , volume=

    Direct preference optimization: Your language model is secretly a reward model , author=. Advances in neural information processing systems , volume=

  28. [35]

    Advances in neural information processing systems , volume=

    Learning to summarize with human feedback , author=. Advances in neural information processing systems , volume=

  29. [36]

    2024 , eprint=

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models , author=. 2024 , eprint=

    2024

  30. [37]

    Guo, Daya and Yang, Dejian and Zhang, Haowei and Song, Junxiao and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Zhang, Ruoyu and Ma, Shirong and Bi, Xiao and Zhang, Xiaokang and Yu, Xingkai and Wu, Yu and Wu, Z. F. and Gou, Zhibin and Shao, Zhihong and Li, Zhuoshu and Gao, Ziyi and Liu, Aixin and Xue, Bing and Wang, Bingxuan and Wu, Bochao and Feng, Bei ...

  31. [38]

    arXiv preprint arXiv:2501.12948 , year=

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year=

    Pith/arXiv arXiv

  32. [40]

    2025 , eprint=

    Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning , author=. 2025 , eprint=

    2025

  33. [41]

    2025 , eprint=

    Qwen3 Technical Report , author=. 2025 , eprint=

    2025

  34. [46]

    5-coder technical report , author=

    Qwen2. 5-coder technical report , author=. arXiv preprint arXiv:2409.12186 , year=

    Pith/arXiv arXiv

  35. [47]

    Advances in neural information processing systems , volume=

    Language models are few-shot learners , author=. Advances in neural information processing systems , volume=

  36. [48]

    2024 , eprint=

    GPT-4 Technical Report , author=. 2024 , eprint=

    2024

  37. [50]

    2025 , eprint=

    Qwen2.5 Technical Report , author=. 2025 , eprint=

    2025

  38. [51]

    arXiv preprint arXiv:2307.09288 , year=

    Llama 2: Open foundation and fine-tuned chat models , author=. arXiv preprint arXiv:2307.09288 , year=

    Pith/arXiv arXiv

  39. [52]

    2024 , eprint=

    Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement , author=. 2024 , eprint=

    2024

  40. [57]

    International Conference on Learning Representations , volume=

    Let's verify step by step , author=. International Conference on Learning Representations , volume=

  41. [59]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Math-shepherd: Verify and reinforce llms step-by-step without human annotations , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  42. [60]

    American Invitational Mathematics Examination (AIME) 2024 , author=

    2024

  43. [61]

    American Invitational Mathematics Examination (AIME) 2025 , author=

    2025

  44. [64]

    2021 , eprint=

    Training Verifiers to Solve Math Word Problems , author=. 2021 , eprint=

    2021

  45. [65]

    arXiv preprint arXiv:2107.03374 , year=

    Evaluating large language models trained on code , author=. arXiv preprint arXiv:2107.03374 , year=

    Pith/arXiv arXiv

  46. [66]

    International Conference on Learning Representations , volume=

    Metamath: Bootstrap your own mathematical questions for large language models , author=. International Conference on Learning Representations , volume=

  47. [68]

    2024 , eprint=

    Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models , author=. 2024 , eprint=

    2024

  48. [70]

    2025 , note=

    DeepScaleR: Surpassing O1-Preview with a 1.5B Model by Scaling RL , author=. 2025 , note=

    2025

  49. [71]

    Notion Blog , volume=

    Deepscaler: Surpassing o1-preview with a 1.5 b model by scaling rl , author=. Notion Blog , volume=

  50. [72]

    Proceedings of the Twentieth European Conference on Computer Systems , pages=

    Hybridflow: A flexible and efficient rlhf framework , author=. Proceedings of the Twentieth European Conference on Computer Systems , pages=

  51. [73]

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher R \'e , and Azalia Mirhoseini. 2024. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787

    Pith/arXiv arXiv 2024

  52. [74]

    Suchanek, and Gaël Varoquaux

    Lihu Chen, Gerard de Melo, Fabian M. Suchanek, and Gaël Varoquaux. 2026. https://arxiv.org/abs/2506.09669 Query-level uncertainty in large language models . Preprint, arXiv:2506.09669

    arXiv 2026

  53. [75]

    Xinghao Chen, Zhijing Sun, Guo Wenjin, Miaoran Zhang, Yanjun Chen, Yirong Sun, Hui Su, Yijie Pan, Dietrich Klakow, Wenjie Li, and 1 others. 2025. Unveiling the key factors for distilling chain-of-thought reasoning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 15094--15119

    2025

  54. [76]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . Preprint, arXiv:2110.14168

    Pith/arXiv arXiv 2021

  55. [77]

    Shrey Desai and Greg Durrett. 2020. Calibration of pre-trained transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 295--302

    2020

  56. [78]

    Yichao Fu, Xuewei Wang, Yuandong Tian, and Jiawei Zhao. 2025. Deep think with confidence. arXiv preprint arXiv:2508.15260

    Pith/arXiv arXiv 2025

  57. [79]

    Yonatan Geifman and Ran El-Yaniv. 2017. Selective classification for deep neural networks. Advances in neural information processing systems, 30

    2017

  58. [80]

    Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. 2024. A survey of confidence estimation and calibration in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6577--6595

    2024

  59. [81]

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International conference on machine learning, pages 1321--1330. PMLR

    2017

  60. [82]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, and 175 others. 2025. https://doi.org/10.1038/s41586-025-09422-z Deepseek-r1 incentivizes reasoning in llms through reinforcement lear...

    doi:10.1038/s41586-025-09422-z 2025

  61. [83]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874

    Pith/arXiv arXiv 2021

  62. [84]

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531

    Pith/arXiv arXiv 2015

  63. [85]

    Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. 2024. V-star: Training verifiers for self-taught reasoners. arXiv preprint arXiv:2402.06457

    Pith/arXiv arXiv 2024

  64. [86]

    Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 9:962--977

    2021

  65. [87]

    Woogyeol Jin, Taywon Min, Yongjin Yang, Swanand Ravindra Kadhe, Yi Zhou, Dennis Wei, Nathalie Baracaldo, and Kimin Lee. 2026. Entropy-aware on-policy distillation of language models. arXiv preprint arXiv:2603.07079

    Pith/arXiv arXiv 2026

  66. [88]

    Abhinav Joshi, Areeb Ahmad, and Ashutosh Modi. 2025. Calibration across layers: Understanding calibration evolution in llms. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 14697--14725

    2025

  67. [89]

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, and 17 others. 2022. https://arxiv.org/abs/2207.05221 Language models (mostly...

    Pith/arXiv arXiv 2022

  68. [90]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199--22213

    2022

  69. [91]

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664

    Pith/arXiv arXiv 2023

  70. [92]

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. 2022. https://arxiv.org/abs/2206.14858 Solving quantitative reasoning problems with language models . Preprint, arXiv:2206.14858

    Pith/arXiv arXiv 2022

  71. [93]

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let's verify step by step. In International Conference on Learning Representations, volume 2024, pages 39578--39601

    2024

  72. [94]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334

    Pith/arXiv arXiv 2022

  73. [95]

    Xiaoou Liu, Tiejin Chen, Longchao Da, Chacha Chen, Zhen Lin, and Hua Wei. 2025. Uncertainty quantification and confidence calibration in large language models: A survey. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 6107--6117

    2025

  74. [96]

    Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica

    Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. 2025. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl. Notion Blog

    2025

  75. [97]

    Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. 2021. Revisiting the calibration of modern neural networks. Advances in neural information processing systems, 34:15682--15694

    2021

  76. [98]

    Cooper, and Milos Hauskrecht

    Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI'15, page 2901–2907. AAAI Press

    2015

  77. [99]

    Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. https://arxiv.org/abs/2112.00114 Show your work: Scratchpads for intermediate computation with language models . Preprint, arXiv:2112.00114

    Pith/arXiv arXiv 2021

  78. [100]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others. 2024. https://arxiv.org/abs/2303.08774 Gpt-4 technical r...

    Pith/arXiv arXiv 2024

  79. [101]

    John Platt. 2000. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Adv. Large Margin Classif., 10

    2000

  80. [102]

    Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, and 24 others. 2025. https://arxiv.org/abs/2412.15115 Qwen2.5 technical report . Preprint, arXiv:2412.15115

    Pith/arXiv arXiv 2025

Showing first 80 references.