Pith.

Machine Learning · arXiv:2610.20401 v1 · submitted 14 Oct 2026 · 65 references

Training Language Models to Fail Gracefully

Daman Arora, Xuecheng Liu, Himanshu Gaurav Singh, Fahim Tajwar, Abitha Thankaraj, Ruslan Salakhutdinov, Haiwen Feng, Yuda Song

show 1 more author

Andrea Zanette

The paper says

“Reinforcement learning makes language models better at solving tasks, but the same mechanism also rewards guessing. When success is the only way to earn reward, opting for a safe fallback is always worse than a long-shot attempt.”

more of the abstract

When a wrong action can corrupt a database or ship broken code, that incentive can be costly. Therefore, we propose training models to fail gracefully: to preserve successful behavior whenever possible while learning to execute a safe fallback policy when acting would be unreliable. However, naively using policy gradients with intermediate reward for fallback behavior is brittle due to unfavorable optimization dynamics. We address this challenge with GracefulRL, which reshapes the optimization landscape to promote graceful failure. Importantly, this behavior emerges directly from outcome-level reward signals, without annotations specifying when to fall back or additional inference-time machinery such as a verifier. Across a range of tasks and scales, we train models that largely preserve their task-solving capability while selectively falling back when acting would be unreliable. More broadly, our results show that reinforcement learning can not only incentivize models to succeed, but also to reason about the limits of their capabilities and change course when they determine that success is out of reach.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 15 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. Worth a serious look: the graceful-failure framing and the log-barrier objective are simple, well-motivated, and reproduce convincingly in a controlled setting, but the LLM results are single-run and the key non-degeneracy assumption is never verified on real policies. This line is wrong

2 major objections · 8 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 65 references say.

The paper says, at Appendix A.4, Assumption 4; Propositions 2 and 5; Figure 1(b)

The review did not quote this passage.

Pith says: Objection 1 of 2 · major

The paper says, at Section 5.2, Table 3, Figures 4 and 6

The review did not quote this passage.

Pith says: Objection 2 of 2 · major

8 minor objections

The paper says, at Sections 3–4 and Appendices A.3–A.4

Pith says: minor objection 1 of 8

The proposition numbering is inconsistent between the main text and the appendix: Section 3's Proposition 1 is proved as Proposition 3 in Appendix A.3, and Section 4's Proposition 2 as Proposition 5 in Appendix A.4; please align the numbering.

The paper says, at Section 5.2.1 and Figure 1 caption

Pith says: minor objection 2 of 8

The c=0 conditioning is described as 'standard RLVR' and 'RL with binary rewards,' but at c=0 the GracefulRL objective reduces to log pS (MaxRL), which is not the expected-reward objective pS + cpF; please state explicitly that the 79.1% baseline is the jointly trained model at its c=0 conditioning level.

show 6 more

The paper says, at Appendix E.3

Pith says: minor objection 3 of 8

The appendix heading promises 'checkpoint selection' for Minesweeper, but no selection criterion is given; please specify whether the final checkpoint was used for the numbers in Figure 7.

The paper says, at Section 5.2

Pith says: minor objection 4 of 8

The LLM experiments appear to be single runs with no variance estimates; please state this explicitly and soften seed-independent language ('consistently Pareto-dominates') for these settings, or add multiple-seed results.

The paper says, at Section 5.2.1 and Figures 12–13

Pith says: minor objection 5 of 8

The Pareto-dominance claim 'across the entire frontier' should be scoped to the measured evaluation points, since GracefulRL's frontier is discrete (five c values) while RLCR's is a continuum of thresholds; if the two fronts cross near the endpoints, the claim as worded is too strong.

The paper says, at Throughout

Pith says: minor objection 6 of 8

Please fix typos and naming inconsistencies: 'it's behavior' in Section 1, 'achieveing' in Appendix C.5, and the DAPO-17k versus DAPO-Math-17K naming mismatch.

The paper says, at Appendix B

Pith says: minor objection 7 of 8

The variance-reduction discussion is conditional on V′(0+)<0 with no argument that this condition holds for the estimator; please state what is actually known about the variance of the smoothed weights.

The paper says, at Figure 2(c)

Pith says: minor objection 8 of 8

Please clarify in the caption of Figure 2(c) that the plotted quantity is the expected environment reward (pS + cpF) rather than the optimized surrogate, since the two methods optimize different objectives.

What would settle it

On the Qwen3-4B checkpoints used for Figure 16 (and the collapsed RLOO or online-reward-tuning checkpoints where pF or pE exceeds roughly 0.9), estimate pS, pF, pE and the conditional score vectors gS, gF, gE from rollouts. Compute ‖gS‖, inf_{λ∈[1−c,1]}‖λgS+(1−λ)gF‖, and the gradient norms ‖∇J_GracefulRL‖ and ‖∇J_PG‖. If the GracefulRL gradient norm decays like (1−p_o) rather than staying bounded away from zero, or if either Assumption 4 direction falls below a positive threshold while the policy is saturated, Proposition 5's guarantee fails in exactly the regime the paper relies on. Repeating at the 1.7B scale would confirm the finding is not scale-specific.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 15 Oct 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 1 min 18 s · 36,617 tokens read, 7,369 written
  3. Editor wrote the verdict above 28 s · 38,180 tokens read, 993 written
  4. Referee major revision: 2 major objections, 8 minor 4 min 5 s · 36,890 tokens read, 28,501 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Here is my read after going through it. The genuinely new piece is the ternary outcome formulation—Success > Fallback > Error—and the claim that outcome-level rewards, without abstention labels or an inference-time verifier, can shape when a model stops trying and falls back. The optimization story is clean: a log-barrier objective over log pS and log(1-pE) avoids the vanishing-gradient trap that makes expected-reward policy gradient collapse near a saturated fallback. I checked the population-level propositions; the algebra is correct under the stated assumptions. The authors are also upfront that the objective is TailRL specialized to ternary rewards plus epsilon smoothing, and that c=0 reduces to MaxRL's log pS. That lineage does not bother me. The contribution is the graceful-failure problem formulation and the demonstration that the objective changes behavior, not a claim of a brand-new objective.

The SafeMaze results are the strongest part: multi-seed, with GRPO/RLOO collapse documented and GracefulRL Pareto-dominating reward-tuning and DPG. That is real evidence the mechanism exists. The LLM experiments are more fragile. All are single runs without error bars, and the headline AIME number compares conditioning suffixes on one jointly-trained model, where c=0 optimizes log pS rather than a separately-trained standard RLVR baseline. MATH500 also doubles as the checkpoint-selection and reported test set. RLCR is actually benchmarked, which is more than many papers do, but the other close abstention-RL baselines are only discussed. None of this is fatal; all of it is addressable with seeds and a cleaner baseline protocol.

The theoretical soft spot is Assumption 4. It does real work in Proposition 5, and the only illustration is a three-action softmax. The authors acknowledge that the proofs cover population gradients only (b372), but that is not the main gap: the non-degeneracy condition itself is unverified for the ResNet and Qwen policies. If conditional score directions vanish near saturation in a trained network, the central advantage over vanilla PG is not established. I would not call the paper wrong because of this, but the Θ(1) gradient claim should be read as conditional.

Who gets value: people designing abstention and reliability training for LLM agents, and anyone working on likelihood-based RL objectives. The conditioned single-model reliability frontier is itself a useful trick. I would send it to review, asking for LLM seeds, a cleaner c=0 baseline, and either verification or de-emphasis of Assumption 4.

The strongest objection

The mechanism claimed to make GracefulRL robust is Proposition 5: unlike JPG, whose gradient is O(1−p_o) near a saturated outcome (Proposition 3), ∇J_GracefulRL stays Θ(1) near fallback or error saturation. The lower bound rests entirely on Assumption 4: near pF→1, ‖gS‖≥δF, and near pE→1, inf_{λ∈[1−c,1]}‖λgS+(1−λ)gF‖≥δE. The only illustration is a three-action softmax with one logit per outcome (Appendix A.4), where the bounds follow immediately. For the actual ResNet and Qwen transformer policies, gS is a trajectory-level conditional expectation of sums of token score vectors, and the bound additionally requires the logit Jacobian not to annihilate these directions, as the authors state (b411). Nothing in the experimental sections measures these quantities. If gS→0 under fallback-saturated policies, or if the convex combination vanishes under error-saturated policies, GracefulRL's gradient also vanishes and the central theoretical distinction from standard policy gradient is not established. The authors acknowledge the proofs cover only population gradients (b372), but the unresolved gap is Assumption 4 itself, not the stochastic estimator.

The reader's weakest assumption is also the most load-bearing condition in the paper. The empirical demonstrations are suggestive, and the authors are appropriately candid about acknowledged limitations: online-only training, absent coding/agentic settings, and population-level proofs only (b210, b211, b212, b372). Those do not constitute objections. The unresolved issue is that Proposition 2/5's Θ(1) gradient guarantee — the stated reason GracefulRL avoids the collapse observed for GRPO/RLOO — depends on Assumption 4, which is neither proved for the trained architectures nor checked empirically. A saturated transformer policy could in principle have near-zero conditional score directions, in which case GracefulRL would stall exactly where the paper claims it cannot. This does not refute the empirical AIME or Minesweeper results, which could still hold as empirical findings, but it does mean the central mechanistic claim is conditional in a way the paper does not test. Because the reader's verdict is already CONDITIONAL and identifies the same assumption, my stress-test does not move the verdict; it sharpens the condition under which the paper should be reconsidered. The concrete test above would either verify Assumption 4 on real policies or demonstrate that the theoretical advantage is not realized, settling the concern directly.

The referee's report

If the results hold, the paper makes a useful dual contribution: it identifies a concrete optimization-dynamics failure mode of outcome-level RL with fallback rewards (gradient suppression near saturation) and provides a simple, analyzable surrogate that avoids it, with credible evidence across three quite different settings. The population-level proofs are algebraically correct under their stated assumptions (I checked the gradient identities in Appendix A.1, the O(1−p_o) bound in Proposition 3, and the Θ(1) argument in Proposition 5), and Appendix B's O(ϵ) bias analysis for the smoothing constant is sound. The empirical study is broad, the experimental design is mostly careful (validation-based checkpoint selection before test evaluation, several baselines, a held-out AbstentionBench), and the code is released. Two caveats are load-bearing: the non-degeneracy assumption underpinning the non-vanishing-gradient claim is unverified for the actual trained policies, and the LLM reliability frontiers come from prompt-embedded caution levels that make it hard to separate learned selective fallback from instruction adherence.

Pith's reading of the paper (a machine's summary, not the authors')

This paper tries to establish that reinforcement learning can shape not only how often a model succeeds, but also how it behaves when it is about to fail: a model can be trained to execute a safe fallback — abstaining, reverting a change, or handing off — whenever acting would be unreliable. The central obstacle is that the natural way to encode this preference, rewarding success above fallback above error and maximizing expected reward, is brittle: the policy-gradient signal shrinks to zero near a policy that has collapsed into always falling back or always erring, so training stalls. The paper's answer, GracefulRL, replaces the expected-reward objective with a log-barrier surrogate $(1-c)\log p_S + c\log(1-p_E)$ that keeps a non-vanishing gradient near those saturated policies. Because the behavior emerges from pure outcome-level rewards, no per-example abstention labels and no inference-time verifier or confidence threshold are needed. If correct, the work implies that outcome-level RL can teach a model to reason about the limits of its own ability and change course.

What carries the argument. The load-bearing object is the surrogate objective $$J_{GracefulRL} = (1-c)\log p_S + c\log(1-p_E),$$ a weighted sum of two log-probabilities defined on the ternary outcome simplex over Success, Fallback, and Error. It acts as a logarithmic barrier on the two undesirable boundaries — the error probability approaching one and the success probability approaching zero — so that under the preference ordering Success ≻ Fallback ≻ Error, moving probability mass away from a collapsed policy never loses optimization signal: the logarithmic form cancels the vanishing factor that suppresses the expected-reward gradient. The practical algorithm estimates this objective through inverse-frequency weights on sampled trajectories, with a small smoothing constant $\epsilon$ that trades bias for variance when successes or non-errors are rare.

Core claim. The central claim is that graceful failure — preserving success whenever possible while selectively executing a safe fallback when success is unlikely — can be learned end-to-end from outcome-level rewards alone, provided the optimization objective is chosen to survive the bad optimization regime. The paper proves that the expected-reward objective $J_{PG}=p_S + c p_F$ has gradients of order $O(1-p_o)$ as any outcome probability approaches one (Proposition 1), which explains why standard policy-gradient training collapses to a policy that always falls back and never solves the task. GracefulRL instead maximizes $J_{GracefulRL} = (1-c)\log p_S + c\log(1-p_E)$, whose gradient is $\Theta(1)$ near fallback or error saturation (Propositions 2 and 5), so the optimizer always retains a signal pointing away from the degenerate corners of the outcome simplex. Empirically, on the AIME benchmark, a Qwen3-4B-Base model trained this way reduces its error rate from 79.1% to 8.9% at caution level $c=0.8$, losing only 3.1 points of success relative to standard RLVR, and the same recipe improves or preserves both sides of the success-error tradeoff in maze navigation and multi-turn Minesweeper.

The premise it rests on. The load-bearing premise is the paper's non-degeneracy assumption (Assumption 4 in Appendix A.4): near a policy that is almost always falling back or almost always erring, the conditional score directions used by the gradient must stay bounded away from zero, and the trained network's parameterization must not annihilate them, with the paper illustrating this only on a three-action softmax toy and not verifying it on the ResNet or Qwen policies actually trained. The authors also acknowledge that the proofs cover population-level gradients only, so the practical claims assume the finite-rollout estimator with epsilon smoothing tracks the population objective.

If the paper is right (Pith's inferences, not the authors')

At test time the policy itself decides when to fall back, so no confidence threshold, verifier, or abstention labels are needed: the same trained model, prompted at its five caution levels, traces a reliability frontier that Pareto-dominates the calibration-reward baseline with test-time thresholding on AIME, AMC, JEEBench, and MATH500.

The headline number on AIME at $c=0.8$ is an error-rate reduction from 79.1% to 8.9%, an 88.8% relative drop, while success falls only 3.1 percentage points relative to binary-reward RL.

Fallback behavior transfers out of distribution: a model trained only on math problems raises its abstention F1 from 0.472 at $c=0.2$ to 0.677 at $c=0.8$ on AbstentionBench's unanswerable questions, above the 0.590 scored by an instruction-tuned model.

In sequential settings the fallback reward can act as reward shaping: in Minesweeper, GracefulRL at $c=0.5$ achieves 77.4% success and 6.0% error, beating standard MaxRL's 67.4% success and 17.0% error on both axes, and it concentrates errors away from the late-game turns where standard RL gambles.

The collapse of GRPO and RLOO into always-fallback or always-error policies is structural, traced to the $O(1-p_o)$ gradient suppression near saturation, so the paper argues that entropy bonuses and reward tuning cannot reliably fix it.

If the non-vanishing-gradient result survives real networks, the same ternary surrogate should transfer to other consequential action spaces the authors did not test, such as code agents that revert their own edits or database agents that roll back a transaction, since the formalism is agnostic to the action space.

The conditioning-suffix design trains several caution levels in one model, which suggests a testable extension: interpolating between the suffix prompts or their embeddings might yield intermediate caution levels the model never saw during training.

The gradient analysis is population-level, and the authors acknowledge this; a natural stress test is whether the finite-rollout estimator with epsilon smoothing preserves the constant-order gradient guarantee at small batch sizes, where rare successes make the inverse-frequency weights unstable.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Reinforcement learning makes language models better at solving tasks, but the same mechanism also rewards guessing. When success is the only way to earn reward, opting for a safe fallback is always worse than a long-shot attempt. When a wrong action can corrupt a database or ship broken code, that incentive can be costly. Therefore, we propose training models to fail gracefully: to preserve successful behavior whenever possible while learning to execute a safe fallback policy when acting would be unreliable. However, naively using policy gradients with intermediate reward for fallback behavior is brittle due to unfavorable optimization dynamics. We address this challenge with GracefulRL, which reshapes the optimization landscape to promote graceful failure. Importantly, this behavior emerges directly from outcome-level reward signals, without annotations specifying when to fall back or additional inference-time machinery such as a verifier. Across a range of tasks and scales, we train models that largely preserve their task-solving capability while selectively falling back when acting would be unreliable. More broadly, our results show that reinforcement learning can not only incentivize models to succeed, but also to reason about the limits of their capabilities and change course when they determine that success is out of reach.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-15

The authors' main claim, as the classifier picked it out 0.91

“More broadly, our results show that reinforcement learning can not only incentivize models to succeed, but also to reason about the limits of their capabilities and change course when they determine that success is out of reach.”

How the main claim is stated: asserted 0.99

What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.69 · human participants 0.04 · controlled comparison 0.14 · randomized assignment 0.01 · states quantitative result 0.04 · compares with prior results 0.22 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Reinforcement learning makes language models better at solving tasks, but the same mechanism also rewards guessing. background 1.00 0.10 0.04 asserted 1.00
When success is the only way to earn reward, opting for a safe fallback is always worse than a long-shot attempt. background 0.96 0.08 0.05 asserted 1.00
When a wrong action can corrupt a database or ship broken code, that incentive can be costly. background 0.99 0.07 0.03 asserted 0.72
Therefore, we propose training models to fail gracefully: to preserve successful behavior whenever possible while learning to execute a safe fallback policy when acting would be unreliable. contribution 0.98 0.28 0.33 asserted 0.89
However, naively using policy gradients with intermediate reward for fallback behavior is brittle due to unfavorable optimization dynamics. gap 0.56 0.53 0.03 asserted 1.00
We address this challenge with GracefulRL, which reshapes the optimization landscape to promote graceful failure. contribution 0.51 0.87 0.10 asserted 1.00
Importantly, this behavior emerges directly from outcome-level reward signals, without annotations specifying when to fall back or additional inference-time machinery such as a verifier. method 0.55 0.88 0.12 asserted 1.00
Across a range of tasks and scales, we train models that largely preserve their task-solving capability while selectively falling back when acting would be unreliable. result 0.99 0.96 0.28 asserted 0.98
More broadly, our results show that reinforcement learning can not only incentivize models to succeed, but also to reason about the limits of their capabilities and change course when they determine that success is out of reach. interpretation 0.99 0.90 0.65 asserted 0.99

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.92

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-15: 1 request, 6,809 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (19)
Figure 1 of the paper
Figure 1. Learning to fail gracefully. (a) GracefulRL augments standard RLVR with an explicit fallback behavior, ordered as Success ≻ Fallback ≻ Error. (b) Standard policy gradient (PG) can lose optimization signal near saturated policies. In contrast, the GracefulRL objective retains a non-vanishing gradient. (c) Using models trained with GracefulRL on the AIME benchmark, the most cautious model (c = 0.8) reduces the error rate from 79.1% to 8.9%, an 88.8% relative reduction, while success only drops by 3.1% compared to RL with binary rewards (c = 0). Therefore, we want models that change…
Figure 2 of the paper
Figure 2. A simple 3-class softmax policy. We use logits zS, zF and zE to create a policy π = (pS, pF , pE) = softmax(zS, zF , zE). (a) For initializations with low initial success probabilities, ∇θJpg leads to traps near the fallback and error vertices (pF = 1, pE = 1), (b) however, ∇θJGracefulRL is more robust to initialization leading to higher success and (c) higher reward. 4.1 The GracefulRL algorithm Algorithm 1: GracefulRL Require: Tasks B; rollouts N; policy πθ; verifier f; fallback utility c ∈ (0, 1); smoothing constant ϵ 1: for each task x ∈ B do 2: Sample y1:N ∼ πθ(· | x) 3: fn ←
Figure 3 of the paper
Figure 3. (a) The SafeMaze task requires the agent to reach the Goal, but if it can’t, it should backtrack to Safe. (b) GRPO and RLOO can collapse to a degenerate policy pF = 1 that is, always going to the Safe square. (c) GracefulRL is robust and Pareto-dominates reward-tuning and DPG (across 3 seeds), achieving higher a Success@ 1 rate at a lower Error@ 1 rate. 5.1 SafeMaze: A Controlled Testbed Environment. For our first experimental setting, we introduce SafeMaze, a maze navigation task where an agent can either reach the Goal, which counts as Success, or navigate to a designated Safe…
Figure 4 of the paper
Figure 4. (a) Training for abstention using standard policy gradient (RLOO) can converge to a policy that always abstains, while our method learns a meaningful policy. (b), (c) Reliability frontiers for the trained Qwen3-1.7B-Base and Qwen3-4B-Base models, comparing RLCR+Thresholding and GracefulRL. GracefulRL achieves a higher success rate at a lower error rate. useful fallback behavior when uncertain. Finally, we also investigate the interplay of various hyperparameters, specifically the role of the number of rollouts N, the fallback reward c, and the smoothing parameter ϵ in Appendix C.4…
Figure 5 of the paper
Figure 5. Different behaviors under uncertainty. A model trained with RLVR has no incentive to acknowledge limitations, and guesses when uncertain. But a model trained with GracefulRL verbalizes and acknowledges its lack in capabilities. Full traces appear in Ap￾pendix D.9. Training and evaluation. We train the Qwen3-1.7B-Base and Qwen3-4B-Base (Yang et al., 2025) models on the DAPO-Math-17K dataset (Yu et al., 2025) using on-policy RL for 1,500 steps and 1,000 steps respectively. For evaluation, we use MATH500 (Hendrycks et al., 2021), AIME 2024-2026, AMC 2022-2024, and JEEBench (numeric)…
Figure 6 of the paper
Figure 6. Abstention F1: GracefulRL (orange), Instruct (blue). Out-of-distribution transfer. We evaluate our model trained on math problems on AbstentionBench (Kirichenko et al., 2025), which contains unknown, underspecified, and otherwise unanswer￾able questions from diverse domains such as History, MedicalQA, Geography and MoralQA. We find that the model retains its in￾tended behavior. More cautious suffixes produce more fallbacks on questions that can’t be answered reliably, while optimistic suffixes attempt more responses. The average abstention F1 score rises from 0.472 at c = 0.2 to…
Figure 7 of the paper
Figure 7. Minesweeper results. (a) Success@ 1 and Error@ 1 rates after training the Qwen3-4B￾Instruct-2507 model. In this setting, GracefulRL achieves a lower rate and higher performance compared to RL with no fallback option. (b) Probability of an error on each turn for MaxRL (c = 0) and GracefulRL (c = 0.5). Number of errors significantly increase at the end of the game for MaxRL since the incentive to ‘play it safe’ reduces and risk taking moves offer better expected return. c = 0.5 collapses achieving neither successes nor errors mirroring our findings in the Sections 5.1 and 5.2. Errors…
Figure 8 of the paper
Figure 8. RLOO with fixed fallback rewards on SafeMaze. We measure pS, pF , pE during training for c = 0.1, c = 0.2, and c ≥ 0.3. Empirically, we observe that when c ≪ pS, the policy can lead to no coverage on fallback trajectories, that is, pF → 0. Similarly, when c ≫ pS, the policy can lead to a degenerate policy where it always goes to the Safe square. C SafeMaze Extended Results C.1 SafeMaze details For the SafeMaze environment, we generate a 15 × 15 grid and pretrain the policy using a mixture of Goal-seeking and Safe-square-seeking trajectories. We ensure that the Goal is always…
Figure 9 of the paper
Figure 9. SafeMaze rollout and smoothing sweeps. (a) Reliability frontiers after 5,000 steps for 16, 32, and 64 rollouts. (b) Training reward with 32 rollouts and c = 0.4: larger ϵ helps initially, whereas ϵ = 1 performs best later. The colored bands mark the highest-reward curve in each training interval pointing to the general principle: when bounded by compute, larger ϵ helps in achieveing higher reward. C.5 Smoothing-Parameter ϵ effects As discussed in Section B, ϵ smoothing can lower variance at the cost of some bias
Figure 10 of the paper
Figure 10. Entropy-bonus sweep for GRPO (top) and RLOO (bottom) on
Figure 11 of the paper
Figure 11. MATH500 validation reward across training checkpoints for RLCR (left) and our
Figure 12 of the paper
Figure 12. Per-dataset reliability frontiers for RLCR and GracefulRL (Ours) with Qwen3-1.7B
Figure 13 of the paper
Figure 13. Per-dataset reliability frontiers for RLCR and GracefulRL (Ours) with Qwen3-4B-Base.
Figure 14 of the paper
Figure 14. Effect of ϵ in language-model RL. Training reward for Qwen3-1.7B-Base with fallback reward c = 0.4 over the first 400 optimization steps. Pale lines show raw reward and dark lines show the smoothed reward. 200 400 600 800 1000 Step 0.11 0.12 0.13 0.14 0.15 0.16 0.17 0.18 Accuracy 200 400 600 800 1000 Step 0.06 0.08 0.10 0.12 0.14 Brier Score 200 400 600 800 1000 Step −0.02 0.00 0.02 0.04 0.06 0.08 0.10 Reward
Figure 15 of the paper
Figure 15. Training dynamics of the Qwen3-4B-Base RLCR policy on AIME. Although the Brier
Figure 16 of the paper
Figure 16. Success and error dynamics on AIME. Qwen3-4B-Base trained with Grace￾fulRL, evaluated at c ∈ {0, 0.2, 0.4, 0.6, 0.8}, and MaxRL trained with binary rewards (dark dashed curve). The left and right panels show pS and pE, respectively, as percentages. Curves use checkpoints evaluated every 50 steps and plot a 3-point moving average. D.6 Success and Error Dynamics on AIME
Figure 17 of the paper
Figure 17. Effect of training compute on the AIME reliability frontier. Qwen3-4B-Base evaluated after 200 (1×), 400 (2×), and 800 (4×) training steps. Each marker is for a different c value and the dashed curves fit the resulting frontiers. D.7 Reliability Frontiers with More Training Compute
Figure 18 of the paper
Figure 18. Online reward tuning loses fallback behavior. Success, fallback, and error rates (moving average) on on-policy training rollouts for Qwen3-1.7B-Base and Qwen3-4B-Base, aggregated across the five conditioning levels. frontier moves uniformly upward with more compute: at a given Error@ 1, the policy achieves higher Success@ 1. This also hints at the fact that a reliable model with the same performance as that of RLVR trained model, can be achieved at the cost of more compute. Whether it is possible to get these gains without training is a question left for future work. D.8 Online…
Figure 19 of the paper
Figure 19. Per-dataset abstention performance on AbstentionBench. Abstention F1 across twenty datasets for GracefulRL at c ∈ {0.2, 0.4, 0.6, 0.8} (orange) and Qwen3-4B￾Instruct-2507 (blue). The methods use matching per-dataset sample counts and class balance; the effect of conditioning varies by dataset. D.10 Evaluation on AbstentionBench We evaluate on AbstentionBench (Kirichenko et al., 2025), a benchmark assembled specifically to test whether a model declines to answer when a question cannot be answered as it is posed. The 20 datasets are drawn from various domains spanning grade-school…

Figures are the authors', reproduced from the paper.

References (65)
  1. [1] Bengio, Yoshua and LeCun, Yann , booktitle =. Scaling Learning Algorithms Towards on Pith
  2. [2] Why Language Models Hallucinate , author=. 2025 , eprint= on Pith
  3. [3] Boyd, Stephen and Vandenberghe, Lieven , year=. Convex Optimization , publisher= on Pith
  4. [4] Hinton, Geoffrey E. and Osindero, Simon and Teh, Yee Whye , journal =. A Fast Learning Algorithm for Deep Belief Nets , volume = on Pith
  5. [5] Zhang, Hanning and Diao, Shizhe and Lin, Yong and Fung, Yi and Lian, Qing and Wang, Xingyao and Chen, Yangyi and Ji, Heng and Zhang, Tong. R -Tuning: Instructing Large Language Models to Say ` I Don ' t Know'. Proceedings of the 2024 Conference of the North American Chapter of... on Pithdoi:10.18653/v1/2024.naacl-long.394
  6. [6] Delightful Gradients Accelerate Corner Escape , author=. 2026 , eprint= on Pith
  7. [7] He, Kaiming and Zhang, Xiangyu and Ren, Shaoqing and Sun, Jian , title =. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , month = on Pith
  8. [8] Deep learning , author=. 2016 , publisher= on Pith
  9. [9] OpenAI o1 System Card , author=. 2026 , eprint= on Pith
  10. [10] Guo, Daya and Yang, Dejian and Zhang, Haowei and Song, Junxiao and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Zhang, Ruoyu and Ma, Shirong et al , year=. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning , volume=. Nature , publisher=. doi:10.1038/s4... on Pithdoi:10.1038/s41586-025-09422-z
  11. [11] Tulu 3: Pushing Frontiers in Open Language Model Post-Training , author=. Second Conference on Language Modeling , year= on Pith
  12. [12] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models , author=. 2024 , eprint= on Pith
  13. [13] DeepCoder: A Fully Open-Source 14B Coder at O3-mini Level , author= on Pith
  14. [14] Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedback , author=. 2024 , eprint= on Pith
  15. [15] Let's Verify Step by Step , author =. The Twelfth International Conference on Learning Representations , year = on Pith
  16. [16] Arora, Daman and Singh, Himanshu and Mausam. Have LLM s Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. doi:10.18653/v1/2023.emnlp-main.468 on Pithdoi:10.18653/v1/2023.emnlp-main.468
  17. [17] Vanishing Gradients in Reinforcement Finetuning of Language Models , author=. The Twelfth International Conference on Learning Representations , year= on Pith
  18. [18] Chow, C. , journal=. On optimum recognition error and reject tradeoff , year= on Pith
  19. [19] Ran El-Yaniv and Yair Wiener , title =. Journal of Machine Learning Research , year = on Pith
  20. [20] Liu, Ziyin and Wang, Zhikang and Liang, Paul Pu and Salakhutdinov, Russ R and Morency, Louis-Philippe and Ueda, Masahito , booktitle =. Deep Gamblers: Learning to Abstain with Portfolio Theory , url = on Pith
show 45 more
  1. [21] Elias Stengel-Eskin and Peter Hase and Mohit Bansal , booktitle=. 2024 , url= on Pith
  2. [22] Jixuan Leng and Chengsong Huang and Banghua Zhu and Jiaxin Huang , booktitle=. Taming Overconfidence in. 2025 , url= on Pith
  3. [23] Teaching Models to Express Their Uncertainty in Words , author=. Transactions on Machine Learning Research , issn=. 2022 , url= on Pith
  4. [24] Tian, Katherine and Mitchell, Eric and Zhou, Allan and Sharma, Archit and Rafailov, Rafael and Yao, Huaxiu and Finn, Chelsea and Manning, Christopher. Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Fee... on Pithdoi:10.18653/v1/2023.emnlp-main.330
  5. [25] Potsawee Manakul and Adian Liusie and Mark Gales , booktitle=. SelfCheck. 2023 , url= on Pith
  6. [26] Miao Xiong and Zhiyuan Hu and Xinyang Lu and YIFEI LI and Jie Fu and Junxian He and Bryan Hooi , booktitle=. Can. 2024 , url= on Pith
  7. [27] Language Models (Mostly) Know What They Know , author=. 2022 , eprint= on Pith
  8. [28] Scalable Best-of-N Selection for Large Language Models via Self-Certainty , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year= on Pith
  9. [29] Javier Garc\'ia and Fernando Fern\'andez , title =. Journal of Machine Learning Research , year = on Pith
  10. [30] Constrained Policy Optimization , author =. Proceedings of the 34th International Conference on Machine Learning , pages =. 2017 , editor = on Pith
  11. [31] Geifman, Yonatan and El-Yaniv, Ran , booktitle =. 2019 , editor = on Pith
  12. [32] Softmax Policy Gradient Methods Can Take Exponential Time to Converge , author =. Proceedings of Thirty Fourth Conference on Learning Theory , pages =. 2021 , editor = on Pith
  13. [33] Alekh Agarwal and Sham M. Kakade and Jason D. Lee and Gaurav Mahajan , title =. Journal of Machine Learning Research , year = on Pith
  14. [34] Dan Hendrycks and Collin Burns and Saurav Kadavath and Akul Arora and Steven Basart and Eric Tang and Dawn Song and Jacob Steinhardt , booktitle=. Measuring Mathematical Problem Solving With the. 2021 , url= on Pith
  15. [35] Zhai, Haotian and Liang, Jingcheng and Kang, Dongyeop. Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RL. Findings of the A ssociation for C omputational L inguistics: ACL 2026. 2026. doi:10.18653/v1/2026.findings-acl.985 on Pithdoi:10.18653/v1/2026.findings-acl.985
  16. [36] Zhepei Wei and Xiao Yang and Kai Sun and Jiaqi Wang and Rulin Shao and Jingxiang Chen and Mohammad Kachuee and Teja Gollapudi and Yiwei Liao and Nicolas SCHEFFER and Rakesh Wanga and Anuj Kumar and Yu Meng and Wen-tau Yih and Xin Luna Dong , booktitle=. Truth. 2026 , url= on Pith
  17. [37] Understanding the Impact of Entropy on Policy Optimization , author =. Proceedings of the 36th International Conference on Machine Learning , pages =. 2019 , editor = on Pith
  18. [38] KARL: Knowledge-Aware Reasoning and Reinforcement Learning for Knowledge-Intensive Visual Grounding , author=. 2026 , eprint= on Pith
  19. [39] Hen Davidov and Nachshon Cohen and Oren Kalinsky and Yaron Fairstein and Guy Kushilevitz and Ram Yazdi and Patrick Rebeschini , booktitle=. Knowing When to Quit: A Principled Framework for Dynamic Abstention in. 2026 , url= on Pith
  20. [40] Jiayun Wu and Jiashuo Liu and Zhiyuan Zeng and Tianyang Zhan and Tianle Cai and Wenhao Huang , booktitle=. Mitigating. 2026 , url= on Pith
  21. [41] TIAR: Trajectory-Informed Advantage Reweighting for LLM Abstention Learning , author=. 2026 , eprint= on Pith
  22. [42] Scaling Open-Ended Reasoning to Predict the Future , author=. 2026 , url= on Pith
  23. [43] GRACE: Reinforcement Learning for Grounded Response and Abstention under Contextual Evidence , author=. 2026 , eprint= on Pith
  24. [44] Williams, Ronald J. , journal =. 1992 , pages =. doi:10.1007/BF00992696 , volume = on Pithdoi:10.1007/bf00992696
  25. [45] Qwen3 Technical Report , author=. 2025 , eprint= on Pith
  26. [46] Miles: Enterprise-Grade Reinforcement Learning for Large-Scale Model Post-Training , author =. 2026 , howpublished = on Pith
  27. [47] Training a Generally Curious Agent , author=. Forty-second International Conference on Machine Learning , year= on Pith
  28. [48] Delightful Policy Gradient , author=. Reinforcement Learning Journal , volume=. 2026 , url= on Pith
  29. [49] Mehul Damani and Isha Puri and Stewart Slocum and Idan Shenfeld and Leshem Choshen and Yoon Kim and Jacob Andreas , booktitle=. Beyond Binary Rewards: Training. 2026 , url= on Pith
  30. [50] Qiying Yu and Zheng Zhang and Ruofei Zhu and Yufeng Yuan and Xiaochen Zuo and Yu Yue and Weinan Dai and Tiantian Fan and Gaohong Liu and Lingjun Liu and Xin Liu and Haibin Lin and Zhiqi Lin and Bole Ma and Guangming Sheng and Yuxuan Tong and Chi Zhang and Mofan Zhang and Wang ... on PithPith's reviewarXiv
  31. [51] Polina Kirichenko and Mark Ibrahim and Kamalika Chaudhuri and Samuel J. Bell , year=. AbstentionBench: Reasoning. 2506.09038 , archivePrefix= on PithPith's reviewarXiv
  32. [52] Proximal Policy Optimization Algorithms , author=. 2017 , eprint= on Pith
  33. [53] Mei, Jincheng and Xiao, Chenjun and Dai, Bo and Li, Lihong and Szepesvari, Csaba and Schuurmans, Dale , booktitle =. Escaping the Gravitational Pull of Softmax , url = on Pith
  34. [54] Tail-Likelihood Reinforcement Learning , author=. 2026 , eprint= on Pith
  35. [55] Ahmadian, Arash and Cremer, Chris and Gall. Back to Basics: Revisiting REINFORCE -Style Optimization for Learning from Human Feedback in LLM s. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024. doi:10.18653/v... on Pithdoi:10.18653/v1/2024.acl-long.662
  36. [56] Maximum Likelihood Reinforcement Learning , author=. Forty-third International Conference on Machine Learning , year= on Pith
  37. [57] Peter L Bartlett and Michael I Jordan and Jon D McAuliffe , title =. Journal of the American Statistical Association , volume =. 2006 , publisher =. doi:10.1198/016214505000000907 , URL = on Pithdoi:10.1198/016214505000000907
  38. [58] Tong Zhang , title =. The Annals of Statistics , number =. 2004 , doi = on Pith
  39. [59] Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones , author=. 2021 , eprint= on Pith
  40. [60] Training Verifiers to Solve Math Word Problems , author =. 2021 , eprint = on Pith
  41. [61] Self-Consistency Improves Chain of Thought Reasoning in Language Models , author =. The Eleventh International Conference on Learning Representations , year = on Pith
  42. [62] HybridFlow: A Flexible and Efficient RLHF Framework , author =. 2024 , journal = on Pith
  43. [63] Yang, Yuqing and Chern, Ethan and Qiu, Xipeng and Neubig, Graham and Liu, Pengfei , booktitle =. Alignment for Honesty , url =. doi:10.52202/079017-2030 , editor = on Pithdoi:10.52202/079017-2030
  44. [64] Sheng, Guangming and Zhang, Chi and Ye, Zilingfeng and Wu, Xibin and Zhang, Wang and Zhang, Ru and Peng, Yanghua and Lin, Haibin and Wu, Chuan , title =. Proceedings of the Twentieth European Conference on Computer Systems , pages =. 2025 , isbn =. doi:10.1145/3689031.3696075 ... on Pithdoi:10.1145/3689031.3696075
  45. [65] Tinus , title =. 2007 , howpublished = on Pith
Cited by 1 Pith paper

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment