Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Fixed rewards for "I don't know" make LLM training either suppress uncertainty or collapse into it; the paper claims a decoupled, dynamically weighted advantage fixes both.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 06:29 UTC pith:UNOKB3IW

load-bearing objection Plausible decoupling fix for uncertainty-aware RL, but the load-bearing DURA formula is a heuristic without sensitivity analysis, and the empirical claims need error bars. the 4 major comments →

arxiv 2601.22648 v2 pith:UNOKB3IW submitted 2026-01-30 cs.AI cs.LG

UCPO: Uncertainty-Aware Policy Optimization

Xianzhou Zeng , Jing Huang , Chunmei Xie , Gongrui Nan , Siye Chen , Mengyu Lu , Weiqi Xiong , Qixuan Zhou
show 4 more authors
Junhao Zhang Qiang Zhu Yadong Li Xingzhong Xu
This is my paper
classification cs.AI cs.LG
keywords uncertainty-aware reinforcement learninglarge language modelsreward hackingadvantage biasternary advantage decouplingdynamic uncertainty reward adjustmentabstentionhallucination mitigation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that LLMs can be taught to say "I don't know" through reinforcement learning without either losing accuracy or degenerating into blanket refusal. It argues that the usual trick—giving uncertain answers a fixed middle reward—is inherently biased: once a model gets good, the group-relative advantage of uncertainty turns negative and honest doubt is punished; when a model is weak, the same reward becomes too attractive and the policy collapses into all-uncertainty reward hacking. UCPO replaces the fixed reward with two coupled mechanisms: Ternary Advantage Decoupling, which normalizes correct/wrong and uncertain rollouts in separate channels, and Dynamic Uncertainty Reward Adjustment, which sets the uncertainty advantage as a gain-scaled projection of the right-answer advantage. Experiments on math and general tasks report that this raises the precision of answered questions while keeping F1 stable. A sympathetic reader would care because a reliable "I don't know" is a prerequisite for trustworthy deployment.

Core claim

On the paper's own terms, the discovery is the diagnosis of "advantage bias" in uncertainty-aware RL: a fixed uncertainty reward sits between right and wrong, but after group normalization its advantage is not neutral—it is negative when right answers dominate and excessive when wrong answers dominate. That single mechanism explains why fixed-reward methods oscillate between overconfidence and avoidance degeneracy. UCPO's fix is structural: compute the advantage of correct/wrong rollouts only among themselves, and define the uncertainty advantage as γ(q)·Âright, a dynamically scaled copy of the right-answer advantage, so uncertainty is rewarded exactly when honesty corrects hallucination and

What carries the argument

Ternary Advantage Decoupling (TAD) partitions each group's rollouts into a deterministic channel (right and wrong) and an uncertainty channel, normalizing each separately so correct answers cannot crush the uncertainty signal. The uncertainty channel's advantage is γ(q)·Âright, anchoring doubt to current reasoning ability. Dynamic Uncertainty Reward Adjustment (DURA) sets γ(q) by the ratio of wrong to uncertain rollouts (an uncertainty-gain term) minus the ratio of right to non-uncertain rollouts (an uncertainty-suppression term), with w=1, so the gain is high early or on hard instances and fades as competence grows. These two formulas together are the load-bearing identity of the method.

Load-bearing premise

The whole benefit rests on the un-derived DURA gain formula γ(q), with w=1; if another functional form or weight were correct, the uncertainty advantage could shrink back to zero or inflate into reward hacking.

What would settle it

Train UCPO on the same benchmarks with w swept from 0 to 2, or with γ(q) frozen to a constant. If any fixed-w or constant-gain variant matches UCPO's PAQ and uncertainty trajectory, the dynamic formula is not the cause; if w=0 reinstates reward hacking or overconfidence, Eq. 5's particular balance is doing the work.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Fixed uncertainty reward tuning becomes unnecessary; UCPO reports stable performance without per-task reward search.
  • Models trained this way convert hallucinations into explicit uncertainty rather than guesses, so reliability on answered questions rises without a collapse in informativeness.
  • The method is framework-agnostic: the paper shows UCPO can be layered onto other policy-optimization algorithms and retain its benefit.
  • Training dynamics should no longer show near-0% or near-100% uncertainty ratios; UCPO reports a gradual rise then plateau.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The DURA gain formula is presented with a plausibility story but no derivation; a natural next test is whether the same balance can be justified from first principles or learned end-to-end, and whether w=1 is optimal.
  • The paper's own conclusion admits that the distribution of right/wrong/uncertain rollouts affects uncertainty learning; an editor's consequence is that the method may need calibration on very different datasets.
  • One could test whether the learned uncertainty is genuine by probing with questions that are impossible versus merely difficult; if UCPO only learns a shallow hedging policy, its uncertainty expression should not track the impossibility of the question.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes UCPO (Uncertainty-Aware Policy Optimization), a RL framework for LLMs that introduces a ternary reward space (right, wrong, uncertain) and claims to eliminate the advantage bias of fixed-uncertainty-reward methods like GRPO-UC. The method has two components: Ternary Advantage Decoupling (TAD), which normalizes deterministic and uncertain rollouts in separate channels (Eqs. 3–4), and Dynamic Uncertainty Reward Adjustment (DURA), which sets the uncertainty advantage to a gain factor γ(q) times the right-answer advantage (Eq. 5). Experiments on Qwen3-8B and Llama-3.1-8B-Instruct across math and general-knowledge benchmarks report PAQ and F1 scores, with ablations and an extension to DAPO in the appendix. The paper argues that UCPO resolves reward hacking and overconfidence by maintaining a stable ternary equilibrium.

Significance. If the proposed mechanism is correct, this is a useful contribution to uncertainty-aware RL for LLMs: the ternary-plot analysis in Fig. 2 clearly illustrates two failure modes of fixed uncertainty rewards (majoritarian suppression and reward hacking), and the idea of decoupling deterministic and uncertain advantage signals is natural. The paper also evaluates on external held-out benchmarks with two base models and demonstrates compatibility with DAPO, which are strengths. However, the central formula of DURA (Eq. 5) is introduced without derivation or sensitivity analysis, and the empirical reporting lacks the statistical grounding needed to support the claim of significant improvement. These issues are load-bearing because the method's novelty and the paper's central claim rest on this formula.

major comments (4)
  1. [§3.2, Eq. (5)] The DURA gain formula γ(q) = [Pw/(Pu+Pw+ε)](1−Pu) − w·[Pr/(Pr+Pw+ε)]Pu is introduced as a hand-designed controller without derivation. The root-cause analysis in §2.2 explains why static rewards cause bias, but it does not logically imply that this particular two-term form, with w=1, eliminates the bias. No sensitivity analysis with respect to w or alternative term structures is provided. This is load-bearing because the entire advantage reweighting mechanism is defined by Eq. (5). The paper should either derive γ(q) from a formal objective (e.g., a desired equilibrium condition on P_r, P_w, P_u) or provide a systematic sensitivity study over w and over reasonable alternative forms. In addition, contribution 2 claims UCPO eliminates the need for reward hyperparameter tuning, but w (and λ, α in Appendix Eqs. 6–7) are effectively free parameters fixed at arbitrary values.
  2. [§4.1, Tables 1–2] The empirical support is statistically weak. The paper reports only the average over three sampled responses per instance, with no standard deviations, confidence intervals, or significance tests. With G=8 and no multiple seeds, differences such as UCPO's PAQ 79.63 vs. GRPO-UC(0.2) 77.66 on Qwen3-8B Math may not be reliable. Moreover, the paper uses F1 as a primary metric alongside PAQ, yet in Table 2 UCPO's average F1 is below several GRPO-UC variants on General Tasks: Qwen3-8B avg F1 70.79 vs. 72.63 (ru=0.5) and 73.78 (ru=0.8); Llama-3.1 avg F1 43.10 vs. 50.96 (ru=0.2). Thus the abstract's claim of 'significantly improving reliability' is not uniformly supported by the paper's own primary metrics. The authors should report full distributions, error bars, and significance tests, and discuss the PAQ/F1 tradeoff explicitly.
  3. [§4.4, Table 3; Appendix A, Eqs. (6)–(7)] The ablation study toggles entire components (TAD, DURA, NTF, LRE) but does not isolate the design choices inside DURA. In particular, the weight w is never varied, and the low-resource extensions introduce λ=0.5 and α=2 with no sensitivity analysis. The DURA-off row in Table 3 actually shows a higher General PAQ (58.96) than the full UCPO (58.58), while the F1 collapses (23.41 vs. 43.10). This means the claimed benefit of DURA is entirely dependent on the F1 metric and on the specific gain form. A sensitivity analysis of w, λ, α and a metric-by-metric breakdown are necessary to establish that DURA, rather than some incidental property of the formula, is responsible for the improved reliability.
  4. [§6 and Appendix C.1] The manuscript itself concedes that the method is not fully understood: Section 6 states that "the distribution ratios of different rollout types potentially influence uncertainty learning, a phenomenon observed in our experiments but not fully explored," and Appendix C.1 says the contribution of LRE is "less definitive" and that the interplay of scaling factors and data diversity "warrants further exploration." Since DURA is explicitly a function of these distribution ratios, this unresolved dependence undermines the mechanistic claim that UCPO resolves the ternary imbalance by design. These limitations should be addressed by targeted experiments (e.g., varying the ratio distribution in a controlled way) rather than left as post-hoc caveats.
minor comments (4)
  1. [Table 1] Header contains typos: "MA TH500" should be "MATH500", and "A verage" should be "Average".
  2. [Figure 9] The legend reads "w/o TAN" but the text and Table 3 refer to TAD; this is likely a typo.
  3. [Eq. (5)] The symbol ε appears in Eq. (5) but is not defined there; it is used in Eq. (3) but should be explicitly stated in the DURA context.
  4. [§4.1] The evaluation protocol says metrics are averaged over "three independent responses" but does not specify whether these are three sampling seeds or three temperature-0.6 samples, or how variance across those samples is handled. Clarify and report individual values or spread.

Circularity Check

0 steps flagged

No significant circularity: UCPO's mechanism and hyperparameters are fixed rather than fitted to evaluation data, and the only overlapping-author citation is non-load-bearing.

full rationale

Walking the derivation chain: Eq. (3) defines the deterministic-channel advantage, Eq. (4) defines the uncertainty advantage as a projection of the right-sample advantage, Eq. (5) defines the DURA gain from per-group rollout ratios, and Eqs. (6)-(7) add low-resource smoothing. The paper's central empirical claims (highest PAQ/F1, stable convergence) are evaluated on external held-out benchmarks (AIME24, AMC, MATH500, Minerva, OlympiadBench, GPQA-Diamond, MMLU-Redux2) and are not implied by construction: although training explicitly rewards uncertainty outputs, the reported F1 trade-offs and comparisons against GRPO-UC with multiple ru values are contingent empirical outcomes. The hyperparameters w=1, λ=0.5, and α=2 are stated as fixed implementation constants, not fitted to the test sets; the absence of sensitivity analysis is an evidential gap (also acknowledged in the Conclusion as an unexplored distribution-ratio effect), but it is not circular. The only overlapping-author citation (NGRPO, Nan et al. 2025) appears in Related Work as a negative-sample method and is not load-bearing for UCPO's derivation. Figure 2(c-d) is a visualization of the proposed equations rather than empirical validation, but the paper does not present that figure as an independent prediction. Hence no claimed derivation reduces to its own inputs.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The paper introduces three free hyperparameters (w, lambda, alpha) that are set by hand and not analyzed for robustness. The DURA formula is an ad hoc heuristic, and the prompt-based assumption that 'uncertainty' tokens reflect true uncertainty is unvalidated. No new physical or conceptual entities are introduced.

free parameters (3)
  • w = 1
    Weighting constant in DURA's uncertainty suppression term (Eq. 5); set without sensitivity analysis.
  • lambda = 0.5
    Batch smoothing fusion weight in LRE (Eq. 6); chosen for balance.
  • alpha = 2
    Scaling factor in tanh mapping (Eq. 7); set to stretch gain range.
axioms (3)
  • standard math The policy gradient theorem and GRPO group-relative normalization are valid
    Relies on Shao et al. 2024.
  • domain assumption Uncertainty responses elicited by the prompt are a faithful measure of model uncertainty
    The paper never validates that outputs containing 'uncertainty' correspond to actual knowledge boundaries.
  • ad hoc to paper The DURA gain formula (Eq. 5) is an adequate controller for ternary equilibrium
    The functional form is introduced without derivation or comparison to alternatives.

pith-pipeline@v1.3.0-alltime-deepseek · 14770 in / 9699 out tokens · 95563 ms · 2026-08-03T06:29:40.719026+00:00 · methodology

0 comments
read the original abstract

The key to building trustworthy large language models (LLMs) lies in endowing them with inherent uncertainty expression capabilities, thereby mitigating overconfident errors in high-stakes applications. However, existing RL paradigms such as GRPO often suffer from Advantage Bias due to binary decision spaces and static uncertainty rewards, inducing either excessive conservatism or overconfidence. To tackle this challenge, this paper unveils the root causes of reward hacking and overconfidence in current RL paradigms incorporating uncertainty-based rewards, based on which we propose the UnCertainty-Aware Policy Optimization (UCPO) framework. UCPO employs Ternary Advantage Decoupling to separate and independently normalize deterministic and uncertain rollouts, thereby eliminating advantage bias. Furthermore, a Dynamic Uncertainty Reward Adjustment mechanism adapts uncertainty weights in real-time according to model evolution and instance difficulty. Experimental results in mathematical reasoning and general tasks demonstrate that UCPO effectively resolves the reward imbalance, significantly improving the reliability of the model beyond their knowledge boundaries.

Figures

Figures reproduced from arXiv: 2601.22648 by Chunmei Xie, Gongrui Nan, Jing Huang, Junhao Zhang, Mengyu Lu, Qiang Zhu, Qixuan Zhou, Siye Chen, Weiqi Xiong, Xianzhou Zeng, Xingzhong Xu, Yadong Li.

Figure 1
Figure 1. Figure 1: Illustration of reward imbalance in uncertainty alignment: static rewards trigger overconfidence or avoidance degeneracy, whereas UCPO stabilizes the policy through adaptive calibration. The first employs Supervised Fine-Tuning (SFT) using in￾structional datasets with explicit abstention labels to facil￾itate imitation learning (Amayuelas et al., 2024; Kapoor et al., 2024). However, the high cost of data s… view at source ↗
Figure 2
Figure 2. Figure 2: The Ternary Imbalance Problem in GRPO-UC (a-b) contrasted with the balanced advantage distribution in UCPO (c-d). Each point in the ternary plots represents a specific combination of Right, Wrong, and Uncertain proportions within a group of G outputs. than being punished for errors, the model is penalized because its cautious choice yields returns below the group’s high expectations, compelling overconfide… view at source ↗
Figure 3
Figure 3. Figure 3: Architecture of the UCPO Framework. isolate deterministic and uncertain signals, thereby elim￾inating semantic interference during advantage estimation. We partition the group of G rollouts into a deterministic set Sdet = {o ∈ Right ∪ Wrong} and an uncertainty set Sunc = {o ∈ Uncertain}. The advantage Aˆ i,t is computed through two independent channels: • Deterministic Channel (oi ∈ Sdet): The advantage is… view at source ↗
Figure 4
Figure 4. Figure 4: Evolution of the uncertainty ratio over training steps, comparing baseline GRPO, the proposed UCPO, and GRPO-UC variants with different reward coefficients ru. • Uncertainty Suppression (Term 2): As the model’s proficiency improves (Pr increases), this term penal￾izes unnecessary avoidance. It effectively raises the competitive bar for choosing the uncertainty path, push￾ing the model to commit to a defini… view at source ↗
Figure 5
Figure 5. Figure 5: Aggregated distribution of Accuracy, Hallucination and Uncertainty across different alignment methods. Proportions are averaged over all datasets independently within the Math & Text Reasoning and General Tasks domains. evolution relative to training steps and uncertainty ratios are provided in the Supplementary Material. 4.3. Main Results [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Impact of rollout group size G on the statistical distribu￾tion of gain γ(q). These extensions collectively allow the framework to main￾tain high feedback intensity and decision robustness even under severe resource constraints. B. Prompting Details {question} Please reason step by step. If confident based on reliable knowledge, provide a clear answer and box it with \\boxed{}. If the question lacks clarit… view at source ↗
Figure 7
Figure 7. Figure 7: Prompt template used in Math & Text Reasoning tasks for open-ended question answering. {question} Please reason step by step. If you are confident based on reliable knowledge, only output the choice letter in the answer field, e.g., answer: C. If the question lacks clarity, exceeds your knowledge, involves speculation, prediction, opinion, or any uncertainty, do not guess. Instead, state your limitation an… view at source ↗
Figure 8
Figure 8. Figure 8: Prompt template designed for General Tasks involving multiple-choice questions [PITH_FULL_IMAGE:figures/full_fig_p011_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Training dynamics of UCPO ablation variants on Llama-3.1-8B-Instruct. illustrates the prompt design for General Tasks, specifically multiple-choice questions, where the model selects the cor￾rect answer from given options. C. Additional Experimental Results C.1. Comparative Analysis of Ablation Study Training Processes The training trajectories in [PITH_FULL_IMAGE:figures/full_fig_p012_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Average performance comparison between GRPO-UC and UCPO (Ours) on Qwen3-8B for Math & Text Reasoning tasks across different training steps. effectively leverages DAPO’s exploration-heavy nature to complement UCPO’s calibration, resulting in a GPQA F1 score of 23.45, which is a substantial improvement over the 17.18 of vanilla UCPO. This synergy confirms that UCPO’s advantages stem from its structural opti… view at source ↗
Figure 11
Figure 11. Figure 11: Average performance comparison between GRPO-UC and UCPO (Ours) on Llama-3.1-8B-Instruct for Math & Text Reasoning tasks across different training steps. 0 50 100 150 200 250 300 Steps 0 5 10 15 20 25 30 Uncertainty Ratio (%) GRPO-UC (ru =0.8) 0 50 100 150 200 250 300 Steps 0 5 10 15 20 25 30 Uncertainty Ratio (%) UCPO (Ours) 66 68 70 72 74 76 78 80 82 84 Score (PAQ / F1) 66 68 70 72 74 76 78 80 82 84 Scor… view at source ↗
Figure 12
Figure 12. Figure 12: Average performance comparison between GRPO-UC and UCPO (Ours) on Qwen3-8B for General tasks across different training steps. 0 50 100 150 200 250 300 Steps 0 10 20 30 40 50 60 Uncertainty Ratio (%) GRPO-UC (ru =0.8) 0 50 100 150 200 250 300 Steps 0 10 20 30 40 50 60 Uncertainty Ratio (%) UCPO (Ours) 40 45 50 55 60 65 Score (PAQ / F1) 40 45 50 55 60 65 Score (PAQ / F1) Uncertainty Ratio PAQ Score F1 Score… view at source ↗
Figure 13
Figure 13. Figure 13: Average performance comparison between GRPO-UC and UCPO (Ours) on Llama-3.1-8B-Instruct for General tasks across different training steps. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Abstention as an Action Can Kill Both the Reward Gradient and the KL Anchor: Collapse Law and Repair for Error-Penalized Reinforcement Learning

    cs.LG 2026-07 conditional novelty 7.0

    A KL-anchored gradient learner trained with an error-penalized abstain action can collapse to total refusal even as its mean reward improves; the repair is to train a mandatory confidence report and abstain only at de...

Reference graph

Works this paper leans on

23 extracted references · 18 linked inside Pith · cited by 1 Pith paper

  1. [2]

    Can ai assistants know what they don’t know?arXiv preprint arXiv:2401.13275,

    Cheng, Q., Sun, T., Liu, X., Zhang, W., Yin, Z., Li, S., Li, L., He, Z., Chen, K., and Qiu, X. Can ai assistants know what they don’t know?arXiv preprint arXiv:2401.13275,

    Pith/arXiv arXiv

  2. [4]

    P., Leang, J

    Gema, A. P., Leang, J. O. J., Hong, G., Devoto, A., Mancino, A. C. M., Saxena, R., He, X., Zhao, Y ., Du, X., Madani, M. R. G., et al. Are we done with mmlu? InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (V olume 1: Long Papers), pp. 5069–5096,

    2025

  3. [5]

    Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Pith/arXiv arXiv

  4. [6]

    A survey on hallucination in large language models: Princi- ples, taxonomy, challenges, and open questions.ArXiv, abs/2311.05232,

    Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., and Liu, T. A survey on hallucination in large language models: Princi- ples, taxonomy, challenges, and open questions.ArXiv, abs/2311.05232,

    Pith/arXiv arXiv

  5. [8]

    Kirichenko, P., Ibrahim, M., Chaudhuri, K., and Bell, S. J. Abstentionbench: Reasoning llms fail on unanswerable questions.arXiv preprint arXiv:2506.09038,

    Pith/arXiv arXiv

  6. [9]

    From system 1 to system 2: A survey of reasoning large lan- guage models.arXiv preprint arXiv:2502.17419,

    Li, Z.-Z., Zhang, D., Zhang, M.-L., Zhang, J., Liu, Z., Yao, Y ., Xu, H., Zheng, J., Wang, P.-J., Chen, X., et al. From system 1 to system 2: A survey of reasoning large lan- guage models.arXiv preprint arXiv:2502.17419,

    Pith/arXiv arXiv

  7. [10]

    S., and Lin, M

    Liu, Z., Chen, C., Li, W., Qi, P., Pang, T., Du, C., Lee, W. S., and Lin, M. Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783, 2025a. Liu, Z., Liu, J., He, Y ., Wang, W., Liu, J., Pan, L., Hu, X., Xiong, S., Huang, J., Hu, J., et al. Part i: Tricks or traps? a deep dive into rl for llm reasoning.arXiv preprint ar...

    Pith/arXiv arXiv

  8. [11]

    Ngrpo: Negative-enhanced group relative policy optimization

    Nan, G., Chen, S., Huang, J., Lu, M., Wang, D., Xie, C., Xiong, W., Zeng, X., Zhou, Q., Li, Y ., et al. Ngrpo: Negative-enhanced group relative policy optimization. arXiv preprint arXiv:2509.18851,

    arXiv

  9. [12]

    Knowrl: Exploring knowledgeable reinforcement learn- ing for factuality.arXiv preprint arXiv:2506.19807,

    Ren, B., Qiao, S., Zheng, D., Chen, H., and Zhang, N. Knowrl: Exploring knowledgeable reinforcement learn- ing for factuality.arXiv preprint arXiv:2506.19807,

    Pith/arXiv arXiv

  10. [13]

    Deepseekmath: Push- ing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y ., Wu, Y ., et al. Deepseekmath: Push- ing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Pith/arXiv arXiv

  11. [14]

    The curious case of hallucinatory (un) answerability: Finding truths in the hidden states of over-confident large language models.arXiv preprint arXiv:2310.11877,

    Slobodkin, A., Goldman, O., Caciularu, A., Dagan, I., and Ravfogel, S. The curious case of hallucinatory (un) answerability: Finding truths in the hidden states of over-confident large language models.arXiv preprint arXiv:2310.11877,

    Pith/arXiv arXiv

  12. [15]

    Tian, K., Mitchell, E., Zhou, A., Sharma, A., Rafailov, R., Yao, H., Finn, C., and Manning, C. D. Just ask for calibra- tion: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. arXiv preprint arXiv:2305.14975,

    Pith/arXiv arXiv

  13. [16]

    Tonmoy, S., Zaman, S. M. M., Jain, V ., Rani, A., Rawte, V ., Chadha, A., and Das, A. A comprehensive survey of hallucination mitigation techniques in large language models.ArXiv, abs/2401.01313,

    Pith/arXiv arXiv

  14. [17]

    Truthrl: Incentivizing truthful llms via reinforcement learning.arXiv preprint arXiv:2509.25760,

    Wei, Z., Yang, X., Sun, K., Wang, J., Shao, R., Chen, S., Kachuee, M., Gollapudi, T., Liao, T., Scheffer, N., et al. Truthrl: Incentivizing truthful llms via reinforcement learning.arXiv preprint arXiv:2509.25760,

    Pith/arXiv arXiv

  15. [18]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Pith/arXiv arXiv

  16. [19]

    Are reasoning models more prone to hallucination?arXiv preprint arXiv:2505.23646,

    Yao, Z., Liu, Y ., Chen, Y ., Chen, J., Fang, J., Hou, L., Li, J., and Chua, T.-S. Are reasoning models more prone to hallucination?arXiv preprint arXiv:2505.23646,

    Pith/arXiv arXiv

  17. [20]

    Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    Yu, Q., Zhang, Z., Zhu, R., Yuan, Y ., Zuo, X., Yue, Y ., Dai, W., Fan, T., Liu, G., Liu, L., et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    Pith/arXiv arXiv

  18. [21]

    A survey of reinforcement learning for large reasoning models.arXiv preprint arXiv:2509.08827,

    Zhang, K., Zuo, Y ., He, B., Sun, Y ., Liu, R., Jiang, C., Fan, Y ., Tian, K., Jia, G., Li, P., et al. A survey of reinforcement learning for large reasoning models.arXiv preprint arXiv:2509.08827,

    Pith/arXiv arXiv

  19. [22]

    The surprising effectiveness of negative reinforcement in llm reasoning.arXiv preprint arXiv:2506.01347,

    Zhu, X., Xia, M., Wei, Z., Chen, W.-L., Chen, D., and Meng, Y . The surprising effectiveness of negative reinforcement in llm reasoning.arXiv preprint arXiv:2506.01347,

    arXiv

  20. [23]

    10 UCPO: Uncertainty-Aware Policy Optimization A. UCPO Extensions for Low-Resource Scenarios In low-resource settings with restricted rollout numbers (e.g., G∈[4,8] ), the sparse rollout space introduces significant statistical bias and high variance. The bias arises from the coarse granularity of the probability estimates; with so few rollouts, the discr...

    2025

  21. [2023]

    T., Nachum, O., Vempala, S

    Kalai, A. T., Nachum, O., Vempala, S. S., and Zhang, E. Why language models hallucinate.arXiv preprint arXiv:2509.04664,

    Pith/arXiv arXiv

  22. [2024]

    Amayuelas, A., Wong, K., Pan, L., Chen, W., and Wang, W

    URL https://artofproblemsolving.com/wiki/ index.php/AIME_Problems_and_Solutions. Amayuelas, A., Wong, K., Pan, L., Chen, W., and Wang, W. Y . Knowledge of knowledge: Exploring known- unknowns uncertainty with large language models. In Findings of the Association for Computational Linguis- tics: ACL 2024, pp. 6416–6432,

    2024

  23. [2025]

    org/abs/2508.01781

    URL https://arxiv. org/abs/2508.01781. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models.arXiv e-prints, pp. arXiv–2407,

    Pith/arXiv arXiv