{"id":"3b7edfce-cb2d-4083-b4ef-12521b809c8a","arxiv_id":"2601.22648","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"UCPO uses decoupled advantage normalization and a dynamic uncertainty reward to train LLMs to express uncertainty, improving answer reliability (PAQ) but sometimes lowering F1.","lead":"This paper introduces a new training method, UCPO, that teaches large language models to say 'I don't know' when they truly don't, instead of guessing. It adjusts the reward for uncertainty on the fly, which the authors say improves reliability without sacrificing too much willingness to answer.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DURA gain formula (Eq. 5) is ad hoc; no sensitivity analysis in w or functional form, so the central claim that UCPO eliminates advantage bias is empirically underdetermined.","rationale":"The reader's weakest assumption precisely identifies the spot where the paper's central mechanism is most underdetermined. Having reviewed the derivation of Eq. 4–5 and the experimental sections, I find no evidence that the specific DURA form was validated beyond a single configuration. The ablation study (Table 3) compares UCPO with and without DURA, showing that removing it causes reward hacking, but does not tell us whether the exact functional form is necessary or merely sufficient. The absence of any sensitivity analysis for w, lambda, alpha is particularly troubling because the paper's second contribution claims 'eliminates the need for exhaustive reward hyperparameter tuning,' yet w, lambda, alpha are set by hand. If UCPO's results are robust to these choices, then the method might work for a broad family of dynamic rewards, which would weaken the claim that the specific formula is the 'root cause' solution; if results are not robust, the central claim of reliability is fragile. Either way, the experiment I propose would settle which. I concur with the reader's conditional verdict: the idea has potential, but the evidence does not yet support the strong wording of the abstract.","tokens_in":14975,"tokens_out":8719,"duration_ms":82901,"concrete_test":"Run the Llama-3.1-8B-Instruct General Tasks experiment (Table 2) with w in {0, 0.5, 1, 2} and with an alternative DURA form gamma'(q) = Pw/(Pw+Pu) (no (1-Pu) scaling), keeping all other hyperparameters and the training seed fixed. If the average PAQ and F1 vary by less than 1 point, DURA is not sensitive to w; if they vary by more than 3 points, the central claim of robust tuning-free calibration fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"UCPO's central novelty is the projection in Eq. 4–5: the uncertainty advantage is set to gamma(q) * A_right, where gamma(q) = [Pw/(Pu+Pw+eps)]*(1-Pu) - w*[Pr/(Pr+Pw+eps)]*Pu with w=1. This functional form is introduced without derivation; the paper only gives a plausibility story. The root-cause analysis in Section 2.2 explains why static rewards cause bias, but does not logically imply that this particular gamma eliminates it. Section 3.2 says DURA 'monitors error rates and confidence', but the two-term formula is a hand-designed heuristic. The ablation (Table 3) only removes DURA entirely; it does not test w != 1 or alternate forms. Furthermore, the LRE extensions (Eq. 6–7) add lambda and alpha, fixed at 0.5 and 2, again without sensitivity analysis. The paper claims 'eliminates the need for exhaustive reward hyperparameter tuning' (contribution 2), but the method has at least three tunable parameters (w, lambda, alpha). If UCPO's performance is sensitive to these, the claim that UCPO robustly resolves reward imbalance is not established; if it is insensitive, then the specific DURA form is not the load-bearing element, and the paper's mechanistic explanation is vacuous. This is load-bearing because the entire method is defined by this formula.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes UCPO (Uncertainty-Aware Policy Optimization), a RL framework for LLMs that introduces a ternary reward space (right, wrong, uncertain) and claims to eliminate the advantage bias of fixed-uncertainty-reward methods like GRPO-UC. The method has two components: Ternary Advantage Decoupling (TAD), which normalizes deterministic and uncertain rollouts in separate channels (Eqs. 3–4), and Dynamic Uncertainty Reward Adjustment (DURA), which sets the uncertainty advantage to a gain factor γ(q) times the right-answer advantage (Eq. 5). Experiments on Qwen3-8B and Llama-3.1-8B-Instruct across math and general-knowledge benchmarks report PAQ and F1 scores, with ablations and an extension to DAPO in the appendix. The paper argues that UCPO resolves reward hacking and overconfidence by maintaining a stable ternary equilibrium.","tokens_in":15358,"tokens_out":4823,"duration_ms":52818,"significance":"If the proposed mechanism is correct, this is a useful contribution to uncertainty-aware RL for LLMs: the ternary-plot analysis in Fig. 2 clearly illustrates two failure modes of fixed uncertainty rewards (majoritarian suppression and reward hacking), and the idea of decoupling deterministic and uncertain advantage signals is natural. The paper also evaluates on external held-out benchmarks with two base models and demonstrates compatibility with DAPO, which are strengths. However, the central formula of DURA (Eq. 5) is introduced without derivation or sensitivity analysis, and the empirical reporting lacks the statistical grounding needed to support the claim of significant improvement. These issues are load-bearing because the method's novelty and the paper's central claim rest on this formula.","major_comments":[{"comment":"The DURA gain formula γ(q) = [Pw/(Pu+Pw+ε)](1−Pu) − w·[Pr/(Pr+Pw+ε)]Pu is introduced as a hand-designed controller without derivation. The root-cause analysis in §2.2 explains why static rewards cause bias, but it does not logically imply that this particular two-term form, with w=1, eliminates the bias. No sensitivity analysis with respect to w or alternative term structures is provided. This is load-bearing because the entire advantage reweighting mechanism is defined by Eq. (5). The paper should either derive γ(q) from a formal objective (e.g., a desired equilibrium condition on P_r, P_w, P_u) or provide a systematic sensitivity study over w and over reasonable alternative forms. In addition, contribution 2 claims UCPO eliminates the need for reward hyperparameter tuning, but w (and λ, α in Appendix Eqs. 6–7) are effectively free parameters fixed at arbitrary values.","section":"§3.2, Eq. (5)"},{"comment":"The empirical support is statistically weak. The paper reports only the average over three sampled responses per instance, with no standard deviations, confidence intervals, or significance tests. With G=8 and no multiple seeds, differences such as UCPO's PAQ 79.63 vs. GRPO-UC(0.2) 77.66 on Qwen3-8B Math may not be reliable. Moreover, the paper uses F1 as a primary metric alongside PAQ, yet in Table 2 UCPO's average F1 is below several GRPO-UC variants on General Tasks: Qwen3-8B avg F1 70.79 vs. 72.63 (ru=0.5) and 73.78 (ru=0.8); Llama-3.1 avg F1 43.10 vs. 50.96 (ru=0.2). Thus the abstract's claim of 'significantly improving reliability' is not uniformly supported by the paper's own primary metrics. The authors should report full distributions, error bars, and significance tests, and discuss the PAQ/F1 tradeoff explicitly.","section":"§4.1, Tables 1–2"},{"comment":"The ablation study toggles entire components (TAD, DURA, NTF, LRE) but does not isolate the design choices inside DURA. In particular, the weight w is never varied, and the low-resource extensions introduce λ=0.5 and α=2 with no sensitivity analysis. The DURA-off row in Table 3 actually shows a higher General PAQ (58.96) than the full UCPO (58.58), while the F1 collapses (23.41 vs. 43.10). This means the claimed benefit of DURA is entirely dependent on the F1 metric and on the specific gain form. A sensitivity analysis of w, λ, α and a metric-by-metric breakdown are necessary to establish that DURA, rather than some incidental property of the formula, is responsible for the improved reliability.","section":"§4.4, Table 3; Appendix A, Eqs. (6)–(7)"},{"comment":"The manuscript itself concedes that the method is not fully understood: Section 6 states that \"the distribution ratios of different rollout types potentially influence uncertainty learning, a phenomenon observed in our experiments but not fully explored,\" and Appendix C.1 says the contribution of LRE is \"less definitive\" and that the interplay of scaling factors and data diversity \"warrants further exploration.\" Since DURA is explicitly a function of these distribution ratios, this unresolved dependence undermines the mechanistic claim that UCPO resolves the ternary imbalance by design. These limitations should be addressed by targeted experiments (e.g., varying the ratio distribution in a controlled way) rather than left as post-hoc caveats.","section":"§6 and Appendix C.1"}],"minor_comments":[{"comment":"Header contains typos: \"MA TH500\" should be \"MATH500\", and \"A verage\" should be \"Average\".","section":"Table 1"},{"comment":"The legend reads \"w/o TAN\" but the text and Table 3 refer to TAD; this is likely a typo.","section":"Figure 9"},{"comment":"The symbol ε appears in Eq. (5) but is not defined there; it is used in Eq. (3) but should be explicitly stated in the DURA context.","section":"Eq. (5)"},{"comment":"The evaluation protocol says metrics are averaged over \"three independent responses\" but does not specify whether these are three sampling seeds or three temperature-0.6 samples, or how variance across those samples is handled. Clarify and report individual values or spread.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: UCPO is a real step forward in a crowded area, but the central mechanism is a plausible heuristic, not a derived result, and the empirical writing oversells it. The TAD idea—normalizing deterministic and uncertain rollouts separately—is the strongest contribution. The ternary plots in Fig. 2 make a clear case that a static ru causes either suppression or reward hacking, and the ablation shows both TAD and DURA matter. That's worth crediting.\n\nThe soft spot is Eq. 5. The gamma formula is introduced with a story, not a derivation, and no sensitivity analysis in w or the term structure. The paper claims to eliminate reward hyperparameter tuning, but it adds w, lambda, alpha, even if some are fixed. With w=1 only, you can't tell whether the specific functional form is load-bearing or whether any reasonable balance would work. The ablation removes DURA entirely, which shows the module matters, but not that this particular gamma is right.\n\nEmpirically, the results are also thinner than they look. Three samples per instance, no error bars or significance tests. The average PAQ gains on Qwen3 are typically a point or two; on Llama-3.1 they're large because GRPO-UC collapses catastrophically. F1 sometimes drops below baselines (e.g., GPQA and MMLU on Llama-3.1), so the 'reliability vs informativeness' trade-off is real but not characterized carefully. The paper's own conclusion admits the distribution ratios influence uncertainty learning without full exploration—that's an honest flag and it should be addressed head-on.\n\nI'd send this to review. The idea is worth testing and the experiments are extensive. But the authors need to either derive gamma from a principle or show robustness to its form, and they need to report variance and significance. The code link in the footnote helps—if the code is actually there, that raises my confidence.","headline":"Plausible decoupling fix for uncertainty-aware RL, but the load-bearing DURA formula is a heuristic without sensitivity analysis, and the empirical claims need error bars.","tokens_in":15834,"tokens_out":3240,"would_cite":true,"duration_ms":37093,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fixed rewards for \"I don't know\" make LLM training either suppress uncertainty or collapse into it; the paper claims a decoupled, dynamically weighted advantage fixes both.","keywords":["uncertainty-aware reinforcement learning","large language models","reward hacking","advantage bias","ternary advantage decoupling","dynamic uncertainty reward adjustment","abstention","hallucination mitigation"],"falsifier":"Train UCPO on the same benchmarks with w swept from 0 to 2, or with γ(q) frozen to a constant. If any fixed-w or constant-gain variant matches UCPO's PAQ and uncertainty trajectory, the dynamic formula is not the cause; if w=0 reinstates reward hacking or overconfidence, Eq. 5's particular balance is doing the work.","tokens_in":14900,"feed_emoji":"🧠","tokens_out":4535,"duration_ms":48025,"temperature":0.7,"pith_summary":"The paper tries to establish that LLMs can be taught to say \"I don't know\" through reinforcement learning without either losing accuracy or degenerating into blanket refusal. It argues that the usual trick—giving uncertain answers a fixed middle reward—is inherently biased: once a model gets good, the group-relative advantage of uncertainty turns negative and honest doubt is punished; when a model is weak, the same reward becomes too attractive and the policy collapses into all-uncertainty reward hacking. UCPO replaces the fixed reward with two coupled mechanisms: Ternary Advantage Decoupling, which normalizes correct/wrong and uncertain rollouts in separate channels, and Dynamic Uncertainty Reward Adjustment, which sets the uncertainty advantage as a gain-scaled projection of the right-answer advantage. Experiments on math and general tasks report that this raises the precision of answered questions while keeping F1 stable. A sympathetic reader would care because a reliable \"I don't know\" is a prerequisite for trustworthy deployment.","feed_headline":"Splitting RL rewards by confidence stops LLM reward hacking","feed_subtitle":"A decoupled uncertainty advantage and a dynamic gain keep models honest without turning them into all-refusal machines.","key_machinery":"Ternary Advantage Decoupling (TAD) partitions each group's rollouts into a deterministic channel (right and wrong) and an uncertainty channel, normalizing each separately so correct answers cannot crush the uncertainty signal. The uncertainty channel's advantage is γ(q)·Âright, anchoring doubt to current reasoning ability. Dynamic Uncertainty Reward Adjustment (DURA) sets γ(q) by the ratio of wrong to uncertain rollouts (an uncertainty-gain term) minus the ratio of right to non-uncertain rollouts (an uncertainty-suppression term), with w=1, so the gain is high early or on hard instances and fades as competence grows. These two formulas together are the load-bearing identity of the method.","core_discovery":"On the paper's own terms, the discovery is the diagnosis of \"advantage bias\" in uncertainty-aware RL: a fixed uncertainty reward sits between right and wrong, but after group normalization its advantage is not neutral—it is negative when right answers dominate and excessive when wrong answers dominate. That single mechanism explains why fixed-reward methods oscillate between overconfidence and avoidance degeneracy. UCPO's fix is structural: compute the advantage of correct/wrong rollouts only among themselves, and define the uncertainty advantage as γ(q)·Âright, a dynamically scaled copy of the right-answer advantage, so uncertainty is rewarded exactly when honesty corrects hallucination and","pith_inferences":["The DURA gain formula is presented with a plausibility story but no derivation; a natural next test is whether the same balance can be justified from first principles or learned end-to-end, and whether w=1 is optimal.","The paper's own conclusion admits that the distribution of right/wrong/uncertain rollouts affects uncertainty learning; an editor's consequence is that the method may need calibration on very different datasets.","One could test whether the learned uncertainty is genuine by probing with questions that are impossible versus merely difficult; if UCPO only learns a shallow hedging policy, its uncertainty expression should not track the impossibility of the question."],"forward_implications":["Fixed uncertainty reward tuning becomes unnecessary; UCPO reports stable performance without per-task reward search.","Models trained this way convert hallucinations into explicit uncertainty rather than guesses, so reliability on answered questions rises without a collapse in informativeness.","The method is framework-agnostic: the paper shows UCPO can be layered onto other policy-optimization algorithms and retain its benefit.","Training dynamics should no longer show near-0% or near-100% uncertainty ratios; UCPO reports a gradual rise then plateau."],"fun_headline_variants":["Decouple RL advantages to end overconfidence","Split RL rewards by confidence to stop hacking","Dynamic uncertainty rewards fix RL bias","Ternary advantage decoupling beats reward hacking","Uncertainty-aware RL keeps models honest"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole benefit rests on the un-derived DURA gain formula γ(q), with w=1; if another functional form or weight were correct, the uncertainty advantage could shrink back to zero or inflate into reward hacking.","fun_headline_variants_meta":{"raw":{"variants":["Decouple RL advantages to end overconfidence","Split RL rewards by confidence to stop hacking","Dynamic uncertainty rewards fix RL bias","Ternary advantage decoupling beats reward hacking","Uncertainty-aware RL keeps models honest"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000201,"raw_usage":{"total_tokens":1182,"prompt_tokens":678,"completion_tokens":504,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":422,"completion_tokens_details":{"reasoning_tokens":451}},"tokens_in":422,"tokens_out":504,"duration_ms":5581,"temperature":1.0,"reasoning_tokens":451,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T06:29:40.719026+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train UCPO on the same benchmarks with w swept from 0 to 2, or with γ(q) frozen to a constant. If any fixed-w or constant-gain variant matches UCPO's PAQ and uncertainty trajectory, the dynamic formula is not the cause; if w=0 reinstates reward hacking or overconfidence, Eq. 5's particular balance is doing the work.","supporting_citations":[],"review_version":1}