Pith. sign in

REVIEW 3 major objections 4 minor 51 references

Error-penalized abstention can train a model to refuse everything, because one saturation factor kills the reward gradient and the KL anchor together.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 00:45 UTC pith:FNKBILTQ

load-bearing objection Genuinely new failure mechanism for error-penalized abstention, with clean theory and sharp predictions; the collapse law is real but narrower than the intro advertises, and the gauge-dependent sign condition is the spot to fix before publication. the 3 major comments →

arxiv 2608.00301 v1 pith:FNKBILTQ submitted 2026-07-31 cs.LG cs.CL

Abstention as an Action Can Kill Both the Reward Gradient and the KL Anchor: Collapse Law and Repair for Error-Penalized Reinforcement Learning

Xujun Che , Yuchen Yuan , Weida Zhao , Chenyang Yu This is my paper
classification cs.LG cs.CL
keywords reinforcement learningabstentionhallucinationKL regularizationpolicy gradienterror-penalized scoringadvantage normalizationconfidence calibration
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Error-penalized scoring rules (+1 for a correct answer, −λ for a wrong one, 0 for abstaining) are increasingly prescribed against hallucination, and a rational agent facing such a rule answers exactly when its correctness probability exceeds Chow's threshold t* = λ/(1+λ). This paper proves that the same rule, given to a KL-anchored gradient learner of the kind RLHF uses, can produce the opposite behavior. When abstention is a discrete action, the reward gradient and the KL anchor's restoring force are throttled by the same gate-saturation factor σ'(v), so they die together as the all-abstain vertex is approached; under explicit conditions, the model drifts toward refusing everything, with its mean training reward rising to zero like 1/t, so the training curve reads as improvement while coverage collapses. The advantage estimator compounds the failure: in the sparse-answer regime, group standardization silently replaces every designed penalty λ with an effective penalty of one, moving the learned threshold from t* to 1/2. The proposed repair moves abstention out of the action space entirely—train a mandatory confidence report with a strictly proper score plus a correctness reward, threshold it only at deployment—so no shared factor can kill the report's gradient and its anchor together.

Core claim

Error-penalized abstention rules, statically optimal for a rational agent, are claimed to be self-defeating for a KL-anchored gradient learner with a shared bounded readout. The load-bearing identity d/dv KL(Bern(σ(v)) ∥ Bern(σ(v0))) = σ′(v)(v − v0) shows the KL anchor's restoring force carries the same saturation factor σ′(v) as the reward gradient. Under explicit conditions (blanket answering loses score, shared bounded readout, forward Bernoulli KL anchor), both forces vanish together as the gate saturates; the model drifts to refusing everything, with mean reward rising to zero like 1/t while coverage collapses. Group-based advantage normalization erases the designed penalty below the kn

What carries the argument

The mechanism is the co-death identity d/dv KL(Bern(σ(v)) ∥ Bern(σ(v0))) = σ′(v)(v − v0), which says the RLHF-style forward Bernoulli KL anchor's restoring force toward the base policy is multiplied by the same gate-saturation factor σ′(v) that throttles the policy-gradient reward term. As the gate logit v falls toward the all-abstain vertex, σ′(v) → 0, so the KL term stops pulling the policy back before the model reaches the vertex; only the reverse KL, with diverging vertex cost, would floor the answer rate. Around this identity the paper builds a finite-time collapse law for the mean training reward (1/t under hypotheses checkable per run), an estimator menu showing group normalization er

Load-bearing premise

The sequence-level collapse theorem assumes answer and refusal content are trained on separate parameters, with the refusal branch initialized at base, so a generic single-decoder Transformer sits outside the theorem's coverage; if real refusal content shares parameters with answering, the KL anchor may tax the answer branch and the exact co-death may not occur as derived.

What would settle it

Run the (+1, −λ, 0) rule on a pool satisfying B1 but give each prompt its own gate logit (no shared bias coordinate); the paper's Proposition 1 predicts convergence to Chow's rule, so observing collapse there would refute the shared-readout mechanism. Alternatively, measure the two gradient norms at a saturated gate: the co-death claim falls if the KL-anchor gradient is not vanishing while the reward gradient is.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the paper is right, deploying the (+1, −λ, 0) rule as a direct RL objective—without moving abstention to a continuous report—will tend to destroy answer coverage on exactly the prompts the rule is meant to protect, while the logged mean reward rises and looks like success.
  • The RLHF KL anchor does not prevent this: because the anchor's gradient carries σ′(v), its restoring force vanishes at the all-abstain vertex; only the reverse KL (not used in practice) provides a genuine floor.
  • Group-standardized advantage estimators (GRPO-style) change the effective objective, not just the step sizes: below a knee at answer rate ≈ 1/G the designed penalty λ is replaced by λeff = 1, shifting the Chow threshold from t* to 1/2 and making the group size, not the penalty, set the abstention rate.
  • The structural repair—training a mandatory confidence report with a strictly proper score plus a correctness reward of weight α > 1—has an interior attracting optimum, so coverage, accuracy, and calibration can improve together, as observed in the paper's simulations and live runs.
  • The paper's analysis implies that reported successes of ternary-reward training at λ = 1 are consistent with the theory: at λ = 1 the estimator's rewriting is the identity and the condition B1 can fail on curated data; where B1 holds, a rising mean reward at ordinary budgets is the plateau's signature, not counter-evidence.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural and cheap test of the estimator theory is to vary only the group size G in a fixed training setup: the knee of the drift law and the plateau answer rate p* ≈ Θ(1/G) should move in the predicted direction even in runs far too short to reach the all-abstain plateau.
  • The paper's Remark 1 suggests the operative variable for collapse is not properness of the score but the existence of a reachable region of zero gradient; if so, the repair's immunity should transfer to any scheme that keeps answering mandatory in training and learns confidence outside the action channel, including post-hoc calibration and conformal abstention—an implication the paper names but do
  • If the branch-separable assumption fails in a real shared decoder (the paper's own open case), the co-death may not occur exactly as derived: the refusal branch's content could drift instead of the answer branch's, so the gate-vs-content KL decomposition the paper uses is a worthwhile diagnostic for production fine-tuning.
  • The collapse law's parameterization-dependence is a subtle caution: the 1/t law holds in the logit parameterization, and the paper shows a nonlinear reparameterization changes the exponent; this suggests that the observable invariant is the co-death of the two gradient norms, not the specific rate.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies error-penalized scoring rules (+1 for correct, −λ for wrong, 0 for abstention) when used as reinforcement-learning rewards with a KL anchor to a base policy. It claims that when abstention is a discrete action, a single gate-saturation factor σ′(v) throttles both the reward gradient and the KL anchor's restoring force; under a set of explicit conditions — including that blanket answering loses score in expectation (B1) and that prompts share a bounded readout — the policy drifts toward refusing everything, with mean training reward rising to zero like 1/t. It further claims that group-based advantage estimators such as GRPO change the decay exponent and, in the sparse-answer regime, effectively replace any designed penalty λ by λ_eff = 1, moving the learned threshold from t* to 1/2. The proposed repair is to train a mandatory confidence report with a Brier-style proper score plus a correctness reward and abstain only at deployment by thresholding the report. The paper supports these claims with exact identities, a finite-time collapse theorem, exact enumeration of the estimator menu, simulations, and live fine-tuning experiments on 1.5B and 7B language models.

Significance. If the main claims hold, the paper is a significant cautionary result for the increasingly common practice of training LLMs with error-penalized abstention rewards. Its strengths are substantial: the identity (1) and the envelope in Theorem 1(i) are clean and internally consistent; the estimator-menu predictions (slope 1/2 above the knee, knee at 1/G, λ-invariance below it) are parameter-free and verified by exact enumeration; the live experiments include careful capability certification and an attribution control that directly measures the joint death of the task gradient and the KL-anchor gradient. The report-level analysis is also a valuable design contribution. However, the paper's central 'collapse law' is conditional on a trajectory-level sign condition that is not a consequence of the advertised three ingredients and is gauge-dependent, and the full-sequence lifting theorem does not cover the generic single-decoder architecture used in the live experiments. These issues require revision before the claims can be accepted at face value.

major comments (3)
  1. [§3, Theorem 1(ii); §1 Contribution 1; §D and §E] The advertised three ingredients are not sufficient for collapse. Theorem 1(ii) adds, on a window, the parameter-space certificate, the 'measured sign' c4(τ) ≤ 0, a tilt bound, and a small-β condition. The sign condition is not implied by (B1) plus a bounded readout plus action-level abstention, and it is gauge-dependent: the reparameterization (w,a,c4,c0) ↦ (−w,−a,−c4,c0+c4) leaves v(s) pointwise unchanged while flipping the sign of c4 (Section E). In the paper's own two-type calibration (q_H=0.9, q_L=0.2, E[q]=0.55 < t*=0.818), choose a bounded readout aligned with profitability, φ≈1 on H and φ≈0 on L, with an overconfident initialization v_H≈v_L. Then ∂_{c4}J ≈ σ′(v)[0.5·0.45·1 + 0.5·(−3.4)·0] > 0, so c4 increases and the flow sharpens toward Chow-selective answering rather than collapsing. Thus the sign input selects the basin; the three ingredients do not. The envelope in Theorem 1(
  2. [§3 'Lifting beyond the frozen gate', Theorem 2; §7 Limitations] The full-sequence lifting theorem does not cover the live full-model experiments. Theorem 2 assumes branch-separable content — answer and refusal content on separate parameters, refusals initialized at base — and an explicit KL-gradient regularity bound C_K < ∞. Part (ii)'s invariance of the refusal branch and stationarity of the collapsed manifold fail when content is shared across branches, and the paper's own Limitations state that a generic single-decoder Transformer sits outside on both counts. The Tier-2 runs (Qwen2.5-1.5B/7B, first-token gate in the model's own vocabulary) are in exactly this open class. This does not invalidate the experiments, but the theorem-level coverage claimed in the abstract and in the 'lifting' paragraph does not extend to the headline live setting. The text should either develop a statement for shared-content architectures or consistently label the live
  3. [§G, Proposition 6 and Table 1] The estimator menu's exact enumeration assumes a population-variance normalizer with denominator G and ε=0, and later production features (ε floor, ratio clipping, token-level losses) are explicitly outside the analysis. For the λ-erasure claim this is reasonable as a stylized model, but the main text's 'silently replaces every designed penalty' should carry the caveat that real GRPO variants include additional mechanisms that are not covered by Proposition 6. This is a scope limitation rather than an error, but it should be prominent in the main text, since practitioners may use the λ_eff = 1 claim as a direct statement about production GRPO.
minor comments (4)
  1. [Table 1] The row labeled 'PG / RLOO all p' is visually garbled and hard to parse; please split the vanilla-PG and RLOO rows or use separate table columns with clearer headers.
  2. [Figure 1(b)] The notation 'G1/β' in the legend is not defined; if it denotes a gain budget depending on anchor strength, please define it explicitly.
  3. [§K] The leaderboard sufficient condition relies on a Laplace-type concentration / sharp-threshold assumption, which the appendix calls 'the least rigorous step in this section.' The main text's 'at least 33 of 36 frontier models satisfy this' should carry that assumption visibly rather than in an appendix footnote.
  4. [§D, Eq. (4)] The 'parameter-space certificate' is introduced in the appendix but referenced in Theorem 1(ii) without a name in the main text; please align the notation so the theorem's hypotheses can be checked against the appendix without cross-referencing.

Circularity Check

0 steps flagged

No significant circularity: the derivation is conditional on stated observables and independently confirmed.

full rationale

The central claims are derived, not fitted. Theorem 1 states an explicit envelope from the identity ∂c0J = -(1 - σ̄)L and integrates it to a 1/t law under checkable window hypotheses (parameter certificate, c4 ≤ 0, tilt bound, small-β condition). Those hypotheses are measurements of the run, not the conclusion L(t)→0, so the theorem does not assume what it predicts. The estimator menu (P1–P5) is obtained by exact enumeration of GRPO's group normalization and is tested against zero-parameter enumerated references and live head-only runs; the λ_eff = 1 erasure is derived from the single-answer group's advantage sign(r)√(G-1), not imposed. The report-level repair follows from the Brier gradient ∂κB = -2E[(c-q̄)∂κc], which contains no action-saturation factor, and is again derived rather than assumed. The paper explicitly scopes its lifting theorem, noting that a generic single-decoder Transformer sits outside the branch-separable cases, and treats the live LM runs as empirical observations in that unproved class; this is an honest scope limitation, not circularity. The skeptic concern that the advertised 'three ingredients' are insufficient without the trajectory-level sign condition c4 ≤ 0, and that this sign is gauge-dependent, is a substantive correctness/scope critique, but it does not amount to a circular derivation: the theorem does not define collapse in terms of c4, nor does it fit L(t) to the conclusion. No load-bearing self-citation chain is present; the estimator and baseline citations are external methods. Therefore there is no circular step to quote, and the score is 0.

Axiom & Free-Parameter Ledger

0 free parameters · 6 axioms · 0 invented entities

No constants are fitted to make the central derivation work; theorem constants are run-dependent tracked quantities, and estimator-menu predictions are parameter-free. No new physical or formal entities are invented; the report channel is a proposed training objective, and lambda_eff=1 is an effective parameter of group normalization, not an entity.

axioms (6)
  • domain assumption (B1): D0 = -∫g ds > 0, equivalently E[q] < t*: blanket answering loses score in expectation.
    Necessary for collapse; if E[q] >= t*, the base policy is profitable and the drift is not toward total abstention. Stated as Standing condition (B1) in Section 2.
  • domain assumption The action-level decision layer is a single Bernoulli gate v_theta(s)=c4*sigma(ws+a)+c0 whose feature phi(s) is bounded in (0,1) and shared across prompts.
    The shared bounded readout is what transmits low-tier attraction to profitable prompts and delays greedy recovery (Proposition 5). Section 2.
  • domain assumption The KL anchor is the forward per-decision Bernoulli KL KL(sigma(v)||sigma(v0)); sequence-level KL reduces to gate KL plus branch terms (Theorem 2).
    The co-death identity Eq. (1) depends on the forward direction; reverse KL would floor the answer rate rather than vanish (Corollary 1).
  • domain assumption Correctness qbar(s) and signal density p(s) are stationary during the analyzed training, with a capability ceiling preserving (B1) in the lifted version.
    If the model could quickly improve q past t* on profitable prompts, the collapse could be interrupted; Theorem 2(iii) adds a capability ceiling or race certificate to preserve the law.
  • domain assumption Theorem 1 trajectory hypotheses: rho <= 1/2, c4 <= 0, tilt bound Lambda <= Lambdabar < 1, parameter-space certificate (4), and small-beta condition (5).
    These are explicit checkable conditions, not derived from (B1); the paper verifies them on every reported run.
  • ad hoc to paper Group-based estimator analysis assumes population-variance normalization with denominator G and epsilon=0.
    The lambda-erasure result (Proposition 6) is derived under this normalization; production estimator features such as epsilon floors and ratio clipping remain outside the analysis and are acknowledged as limitations.

pith-pipeline@v1.3.0-alltime-deepseek · 36752 in / 17626 out tokens · 161800 ms · 2026-08-04T00:45:59.181151+00:00 · methodology

0 comments
read the original abstract

Error-penalized scoring rules ($+1$ for a correct answer, $-\lambda$ for a wrong one, $0$ for abstaining) are increasingly prescribed against hallucination: a rational agent facing such a rule answers exactly when its correctness probability exceeds Chow's threshold $t^\ast=\lambda/(1+\lambda)$. We prove that a KL-anchored gradient learner can do the opposite. When abstention is a discrete action, the reward gradient and the anchor's restoring force are throttled by the same gate-saturation factor and die together: under explicit conditions (among them, blanket answering loses score in expectation and prompts share a bounded readout) the model drifts toward refusing everything, its mean training reward rising to zero like $1/t$ in training time $t$, so the curve reads as improvement while coverage collapses. The advantage estimator compounds the failure: in its sparse-answer regime, group normalization silently replaces every designed penalty with an effective penalty of one, moving the learned threshold from $t^\ast$ to $1/2$. The repair is structural: train a mandatory confidence report with a strictly proper score plus a correctness reward, and abstain only at deployment by thresholding the report. The always-emitted report has no gate to saturate, so no shared factor can kill its reward gradient and its anchor together, and its calibrated optimum is attracting. Simulations confirm every prediction, and experiments on language models at two scales confirm the mechanism live: the rule silences questions the models demonstrably still solve within ten optimizer steps, an ablation isolates the cause, and report-level training raises coverage, accuracy, and calibration together.

Figures

Figures reproduced from arXiv: 2608.00301 by Chenyang Yu, Weida Zhao, Xujun Che, Yuchen Yuan.

Figure 1
Figure 1. Figure 1: The geometry of the failure. (a) A hard threshold [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The menu by exact enumeration (q¯ = 0.3, λ = 4.5). (a) GRPO drift ∝ √p above the knee p = 1/G (dotted), ∝ p below; vanilla PG ∝ p throughout. (b) Sparse regime: drift/p collapses onto (2¯q − 1)√ G − 1 for every nominal λ. (c) Exponents converge to 0.5 and 1.0. p ≪ 1/G: a group contains at most one answer, whose advantage is sign(r) √ G − 1 independently of |r|, so the drift is p (2¯q − 1)√ G − 1: the desig… view at source ↗
Figure 3
Figure 3. Figure 3: (a) The invariant L for three anchor strengths; the upturn is ttilt. (b) The proximal surrogate floors the reward; true KL does not. (c) Greedy coverage at λ = 4.5: report level (solid) near-rational; action level (dashed) deploys nothing. optimal action-level report-level signal U ∗ cov U cov U cov weak 0.005 0.035 0.000 0.000 0.005 0.034 medium 0.074 0.259 0.000 0.000 0.074 0.246 strong 0.199 0.469 0.000… view at source ↗
Figure 5
Figure 5. Figure 5: (a) Terminal policy logits at T = 104 , µ = 1, λ = 4.5: with a bounded readout v(s) < 0 for every s (greedy abstains on everything); with an unbounded feature the logit crosses zero and greedy answers a tail. (b) The tilt factor required for escape diverges for unbounded features and is capped for bounded ones. 2 3 4 5 6 penalty λ 0.50 0.75 1.00 1.25 1.50 1.75 2.00 2.25 s elf-k n o wle d g e q u ality μ no… view at source ↗
Figure 6
Figure 6. Figure 6: A horizon-free view: tenter is the first time greedy coverage falls below 2% of rational, texit the first time there￾after it exceeds 20%. (a) log10 texit under true KL with a bounded head at β = 10−3 ; green cells never collapse. (b) Plateau length texit/tenter explodes past a critical λ that increases with signal quality µ, non-monotonically; the ridge is displayed rather than smoothed. then ∂vJβ = σ ′ (… view at source ↗
Figure 7
Figure 7. Figure 7: Rate saturation above the knee: the population [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: The fixed point that group normalization creates. [PITH_FULL_IMAGE:figures/full_fig_p016_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Conservation of collapse per rollout. (a) Against [PITH_FULL_IMAGE:figures/full_fig_p017_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: (a) Boundary versus interior: the action-level [PITH_FULL_IMAGE:figures/full_fig_p017_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Horizon-free phase diagram over (µ, λ). Colour is log10 of the time to competence, the first time the greedily deployed policy attains half of U ∗ ; black cells never be￾come competent within 106 . Top: action-level abstention un￾der true policy KL. Bottom: report-level abstention. Both greedy, identical bounded heads; the two rows’ β multiply different objects (policy KL above, weight-space proximal term… view at source ↗
Figure 12
Figure 12. Figure 12: (a) Each model’s score line acc − λ err; its zero is λcrit = acc/err, and at the deployed λ = 1 the sign of the score is the published index. (b) The same fact in the (conditional accuracy, λ) plane: a negative index implies collapse drift; a positive one does not preclude it. which scores +1/ − 1/0 over more than 36 frontier mod￾els, only three scored above zero at the launch snapshot: at least 33 satisf… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

51 extracted references · 20 linked inside Pith

  1. [1]

    and Zhang, Edwin , year =

    Kalai, Adam Tauman and Nachum, Ofir and Vempala, Santosh S. and Zhang, Edwin , year =. Why Language Models Hallucinate , eprint =

  2. [2]

    , year =

    Kalai, Adam Tauman and Vempala, Santosh S. , year =. Calibrated Language Models Must Hallucinate , eprint =

  3. [3]

    Beyond Binary Rewards: Training

    Damani, Mehul and Puri, Isha and Slocum, Stewart and others , year =. Beyond Binary Rewards: Training. 2507.16806 , archivePrefix =

    Pith/arXiv arXiv

  4. [4]

    2509.25760 , archivePrefix =

    Wei, Zhepei and Yang, Xiao and Sun, Kai and Wang, Jiaqi and others , year =. 2509.25760 , archivePrefix =

    Pith/arXiv arXiv

  5. [5]

    2605.25850 , archivePrefix =

    Pan, Muyu and Zhao, Shu and Zhang, Nan and Shin, Philip and others , year =. 2605.25850 , archivePrefix =

    Pith/arXiv arXiv

  6. [6]

    2604.22779 , archivePrefix =

    Gao, Cheng and Huang, Cheng and Luo, Kangyang and Qiao, Ziqing and others , year =. 2604.22779 , archivePrefix =

    Pith/arXiv arXiv

  7. [7]

    2601.22648 , archivePrefix =

    Zeng, Xianzhou and Huang, Jing and Xie, Chunmei and Nan, Gongrui and others , year =. 2601.22648 , archivePrefix =

    Pith/arXiv arXiv

  8. [8]

    Abstain-

    Zhai, Skylar and Liang, Jingcheng and Kang, Dongyeop , year =. Abstain-. 2604.17073 , archivePrefix =

    Pith/arXiv arXiv

  9. [9]

    Enhancing Reliability across Short and Long-Form

    Wang, Yudong and Yang, Zhe and Ma, Wenhan and Sui, Zhifang and others , year =. Enhancing Reliability across Short and Long-Form. 2512.08944 , archivePrefix =

    arXiv

  10. [10]

    Rejection Improves Reliability: Training

    Xu, Hongshen and Zhu, Zichen and Zhang, Situo and Ma, Da and others , year =. Rejection Improves Reliability: Training. 2403.18349 , archivePrefix =

    Pith/arXiv arXiv

  11. [11]

    The Hallucination Tax of Reinforcement Finetuning , eprint =

    Song, Linxin and Shi, Taiwei and Zhao, Jieyu , year =. The Hallucination Tax of Reinforcement Finetuning , eprint =

  12. [12]

    2509.17730 , archivePrefix =

    Zhang, Bonan and Chen, Zhongqi and Song, Bowen and others , year =. 2509.17730 , archivePrefix =

    arXiv

  13. [13]

    Knowledge-Level Consistency Reinforcement Learning: Dual-Fact Alignment for Long-Form Factuality , eprint =

    Li, Junliang and Wang, Yucheng and Chen, Yan and others , year =. Knowledge-Level Consistency Reinforcement Learning: Dual-Fact Alignment for Long-Form Factuality , eprint =

  14. [14]

    2505.13529 , archivePrefix =

    Yang, Junxiao and Tu, Jinzhe and Liu, Haoran and others , year =. 2505.13529 , archivePrefix =

    arXiv

  15. [15]

    Vanishing Gradients in Reinforcement Finetuning of Language Models , eprint =

    Razin, Noam and Zhou, Hattie and Saremi, Omid and Thilak, Vimal and others , year =. Vanishing Gradients in Reinforcement Finetuning of Language Models , eprint =

  16. [16]

    2020 , title =

    Mei, Jincheng and Xiao, Chenjun and Szepesv. 2020 , title =. 2005.06392 , archivePrefix =

    Pith/arXiv arXiv 2020

  17. [17]

    The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models , eprint =

    Cui, Ganqu and Zhang, Yuchen and Chen, Jiacheng and others , year =. The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models , eprint =

  18. [18]

    Understanding the Effects of

    Kirk, Robert and Mediratta, Ishita and Nalmpantis, Christoforos and Luketina, Jelena and others , year =. Understanding the Effects of. 2310.06452 , archivePrefix =

    Pith/arXiv arXiv

  19. [19]

    A Theory of Regularized

    Geist, Matthieu and Scherrer, Bruno and Pietquin, Olivier , year =. A Theory of Regularized. 1901.11275 , archivePrefix =

    Pith/arXiv arXiv 1901

  20. [20]

    Leverage the Average: an Analysis of

    Vieillard, Nino and Kozuno, Tadashi and Scherrer, Bruno and Pietquin, Olivier and others , year =. Leverage the Average: an Analysis of. 2003.14089 , archivePrefix =

    Pith/arXiv arXiv 2003

  21. [21]

    2402.03300 , archivePrefix =

    Shao, Zhihong and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and others , year =. 2402.03300 , archivePrefix =

    Pith/arXiv arXiv

  22. [22]

    Understanding

    Liu, Zichen and Chen, Changyu and Li, Wenjun and Qi, Penghui and others , year =. Understanding. 2503.20783 , archivePrefix =

    Pith/arXiv arXiv

  23. [23]

    2024 , title =

    Ahmadian, Arash and Cremer, Chris and Gall. 2024 , title =. 2402.14740 , archivePrefix =

    Pith/arXiv arXiv 2024

  24. [24]

    2503.14476 , archivePrefix =

    Yu, Qiying and Zhang, Zheng and Zhu, Ruofei and Yuan, Yufeng and others , year =. 2503.14476 , archivePrefix =

    Pith/arXiv arXiv

  25. [25]

    2501.03262 , archivePrefix =

    Hu, Jian and Liu, Jason Klein and Xu, Haotian and Shen, Wei , year =. 2501.03262 , archivePrefix =

    Pith/arXiv arXiv

  26. [26]

    Scaling Laws for Reward Model Overoptimization , eprint =

    Gao, Leo and Schulman, John and Hilton, Jacob , year =. Scaling Laws for Reward Model Overoptimization , eprint =

  27. [27]

    Skalse, Joar and Howe, Nikolaus H. R. and Krasheninnikov, Dmitrii and Krueger, David , year =. Defining and Characterizing Reward Hacking , eprint =

  28. [28]

    and Martic, Miljan and others , year =

    Christiano, Paul and Leike, Jan and Brown, Tom B. and Martic, Miljan and others , year =. Deep Reinforcement Learning from Human Preferences , eprint =

  29. [29]

    and others , year =

    Stiennon, Nisan and Ouyang, Long and Wu, Jeff and Ziegler, Daniel M. and others , year =. Learning to Summarize from Human Feedback , eprint =

  30. [30]

    Training Language Models to Follow Instructions with Human Feedback , eprint =

    Ouyang, Long and Wu, Jeff and Jiang, Xu and Almeida, Diogo and others , year =. Training Language Models to Follow Instructions with Human Feedback , eprint =

  31. [31]

    Chow, C. K. , year =. On Optimum Recognition Error and Reject Tradeoff , journal =

  32. [32]

    On the Foundations of Noise-free Selective Classification , journal =

    El-Yaniv, Ran and Wiener, Yair , year =. On the Foundations of Noise-free Selective Classification , journal =

  33. [33]

    Selective Classification for Deep Neural Networks , eprint =

    Geifman, Yonatan and El-Yaniv, Ran , year =. Selective Classification for Deep Neural Networks , eprint =

  34. [34]

    Selective Question Answering under Domain Shift , eprint =

    Kamath, Amita and Jia, Robin and Liang, Percy , year =. Selective Question Answering under Domain Shift , eprint =

  35. [35]

    Out-of-Distribution Detection and Selective Generation for Conditional Language Models , eprint =

    Ren, Jie and Luo, Jiaming and Zhao, Yao and Krishna, Kundan and others , year =. Out-of-Distribution Detection and Selective Generation for Conditional Language Models , eprint =

  36. [36]

    Language Models (Mostly) Know What They Know , eprint =

    Kadavath, Saurav and Conerly, Tom and Askell, Amanda and Henighan, Tom and others , year =. Language Models (Mostly) Know What They Know , eprint =

  37. [37]

    Teaching Models to Express Their Uncertainty in Words , eprint =

    Lin, Stephanie and Hilton, Jacob and Evans, Owain , year =. Teaching Models to Express Their Uncertainty in Words , eprint =

  38. [38]

    Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback , eprint =

    Tian, Katherine and Mitchell, Eric and Zhou, Allan and Sharma, Archit and others , year =. Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback , eprint =

  39. [39]

    Reported Confidence in

    Kumaran, Dharshan and others , year =. Reported Confidence in. 2606.29490 , archivePrefix =

    Pith/arXiv arXiv

  40. [40]

    , year =

    Guo, Chuan and Pleiss, Geoff and Sun, Yu and Weinberger, Kilian Q. , year =. On Calibration of Modern Neural Networks , eprint =

  41. [41]

    Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation , eprint =

    Kuhn, Lorenz and Gal, Yarin and Farquhar, Sebastian , year =. Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation , eprint =

  42. [42]

    Mitigating

    Abbasi Yadkori, Yasin and Kuzborskij, Ilja and Stutz, David and others , year =. Mitigating. 2405.01563 , archivePrefix =

    Pith/arXiv arXiv

  43. [43]

    2412.15115 , archivePrefix =

    2024 , title =. 2412.15115 , archivePrefix =

    Pith/arXiv arXiv 2024

  44. [44]

    and Zettlemoyer, Luke , year =

    Joshi, Mandar and Choi, Eunsol and Weld, Daniel S. and Zettlemoyer, Luke , year =. 1705.03551 , archivePrefix =

    Pith/arXiv arXiv

  45. [45]

    When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories , eprint =

    Mallen, Alex and Asai, Akari and Zhong, Victor and Das, Rajarshi and others , year =. When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories , eprint =

  46. [46]

    2511.13029 , archivePrefix =

    Jackson, Declan and Keating, William and Cameron, George and Hill-Smith, Micah , year =. 2511.13029 , archivePrefix =

    arXiv

  47. [47]

    , year =

    Brier, Glenn W. , year =. Verification of Forecasts Expressed in Terms of Probability , journal =

  48. [48]

    , year =

    Savage, Leonard J. , year =. Elicitation of Personal Probabilities and Expectations , journal =

  49. [49]

    , year =

    Gneiting, Tilmann and Raftery, Adrian E. , year =. Strictly Proper Scoring Rules, Prediction, and Estimation , journal =

  50. [50]

    , year =

    Williams, Ronald J. , year =. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning , journal =

  51. [51]

    Approximating

    Schulman, John , year =. Approximating