Machine Learning · arXiv:2610.20401 v1 · submitted 14 Oct 2026 · 65 references
Training Language Models to Fail Gracefully
The paper says
“Reinforcement learning makes language models better at solving tasks, but the same mechanism also rewards guessing. When success is the only way to earn reward, opting for a safe fallback is always worse than a long-shot attempt.”
more of the abstract
When a wrong action can corrupt a database or ship broken code, that incentive can be costly. Therefore, we propose training models to fail gracefully: to preserve successful behavior whenever possible while learning to execute a safe fallback policy when acting would be unreliable. However, naively using policy gradients with intermediate reward for fallback behavior is brittle due to unfavorable optimization dynamics. We address this challenge with GracefulRL, which reshapes the optimization landscape to promote graceful failure. Importantly, this behavior emerges directly from outcome-level reward signals, without annotations specifying when to fall back or additional inference-time machinery such as a verifier. Across a range of tasks and scales, we train models that largely preserve their task-solving capability while selectively falling back when acting would be unreliable. More broadly, our results show that reinforcement learning can not only incentivize models to succeed, but also to reason about the limits of their capabilities and change course when they determine that success is out of reach.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 15 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. Worth a serious look: the graceful-failure framing and the log-barrier objective are simple, well-motivated, and reproduce convincingly in a controlled setting, but the LLM results are single-run and the key non-degeneracy assumption is never verified on real policies. This line is wrong
2 major objections · 8 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 65 references say.
The paper says, at Appendix A.4, Assumption 4; Propositions 2 and 5; Figure 1(b)
The review did not quote this passage.
Pith says: Objection 1 of 2 · major
The paper says, at Section 5.2, Table 3, Figures 4 and 6
The review did not quote this passage.
Pith says: Objection 2 of 2 · major
8 minor objections
The paper says, at Sections 3–4 and Appendices A.3–A.4
Pith says: minor objection 1 of 8
The proposition numbering is inconsistent between the main text and the appendix: Section 3's Proposition 1 is proved as Proposition 3 in Appendix A.3, and Section 4's Proposition 2 as Proposition 5 in Appendix A.4; please align the numbering.
The paper says, at Section 5.2.1 and Figure 1 caption
Pith says: minor objection 2 of 8
The c=0 conditioning is described as 'standard RLVR' and 'RL with binary rewards,' but at c=0 the GracefulRL objective reduces to log pS (MaxRL), which is not the expected-reward objective pS + cpF; please state explicitly that the 79.1% baseline is the jointly trained model at its c=0 conditioning level.
show 6 more
The paper says, at Appendix E.3
Pith says: minor objection 3 of 8
The appendix heading promises 'checkpoint selection' for Minesweeper, but no selection criterion is given; please specify whether the final checkpoint was used for the numbers in Figure 7.
The paper says, at Section 5.2
Pith says: minor objection 4 of 8
The LLM experiments appear to be single runs with no variance estimates; please state this explicitly and soften seed-independent language ('consistently Pareto-dominates') for these settings, or add multiple-seed results.
The paper says, at Section 5.2.1 and Figures 12–13
Pith says: minor objection 5 of 8
The Pareto-dominance claim 'across the entire frontier' should be scoped to the measured evaluation points, since GracefulRL's frontier is discrete (five c values) while RLCR's is a continuum of thresholds; if the two fronts cross near the endpoints, the claim as worded is too strong.
The paper says, at Throughout
Pith says: minor objection 6 of 8
Please fix typos and naming inconsistencies: 'it's behavior' in Section 1, 'achieveing' in Appendix C.5, and the DAPO-17k versus DAPO-Math-17K naming mismatch.
The paper says, at Appendix B
Pith says: minor objection 7 of 8
The variance-reduction discussion is conditional on V′(0+)<0 with no argument that this condition holds for the estimator; please state what is actually known about the variance of the smoothed weights.
The paper says, at Figure 2(c)
Pith says: minor objection 8 of 8
Please clarify in the caption of Figure 2(c) that the plotted quantity is the expected environment reward (pS + cpF) rather than the optimized surrogate, since the two methods optimize different objectives.
What would settle it
On the Qwen3-4B checkpoints used for Figure 16 (and the collapsed RLOO or online-reward-tuning checkpoints where pF or pE exceeds roughly 0.9), estimate pS, pF, pE and the conditional score vectors gS, gF, gE from rollouts. Compute ‖gS‖, inf_{λ∈[1−c,1]}‖λgS+(1−λ)gF‖, and the gradient norms ‖∇J_GracefulRL‖ and ‖∇J_PG‖. If the GracefulRL gradient norm decays like (1−p_o) rather than staying bounded away from zero, or if either Assumption 4 direction falls below a positive threshold while the policy is saturated, Proposition 5's guarantee fails in exactly the regime the paper relies on. Repeating at the 1.7B scale would confirm the finding is not scale-specific.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 15 Oct 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 2 major objections, 8 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Here is my read after going through it. The genuinely new piece is the ternary outcome formulation—Success > Fallback > Error—and the claim that outcome-level rewards, without abstention labels or an inference-time verifier, can shape when a model stops trying and falls back. The optimization story is clean: a log-barrier objective over log pS and log(1-pE) avoids the vanishing-gradient trap that makes expected-reward policy gradient collapse near a saturated fallback. I checked the population-level propositions; the algebra is correct under the stated assumptions. The authors are also upfront that the objective is TailRL specialized to ternary rewards plus epsilon smoothing, and that c=0 reduces to MaxRL's log pS. That lineage does not bother me. The contribution is the graceful-failure problem formulation and the demonstration that the objective changes behavior, not a claim of a brand-new objective.
The SafeMaze results are the strongest part: multi-seed, with GRPO/RLOO collapse documented and GracefulRL Pareto-dominating reward-tuning and DPG. That is real evidence the mechanism exists. The LLM experiments are more fragile. All are single runs without error bars, and the headline AIME number compares conditioning suffixes on one jointly-trained model, where c=0 optimizes log pS rather than a separately-trained standard RLVR baseline. MATH500 also doubles as the checkpoint-selection and reported test set. RLCR is actually benchmarked, which is more than many papers do, but the other close abstention-RL baselines are only discussed. None of this is fatal; all of it is addressable with seeds and a cleaner baseline protocol.
The theoretical soft spot is Assumption 4. It does real work in Proposition 5, and the only illustration is a three-action softmax. The authors acknowledge that the proofs cover population gradients only (b372), but that is not the main gap: the non-degeneracy condition itself is unverified for the ResNet and Qwen policies. If conditional score directions vanish near saturation in a trained network, the central advantage over vanilla PG is not established. I would not call the paper wrong because of this, but the Θ(1) gradient claim should be read as conditional.
Who gets value: people designing abstention and reliability training for LLM agents, and anyone working on likelihood-based RL objectives. The conditioned single-model reliability frontier is itself a useful trick. I would send it to review, asking for LLM seeds, a cleaner c=0 baseline, and either verification or de-emphasis of Assumption 4.
The strongest objection
The mechanism claimed to make GracefulRL robust is Proposition 5: unlike JPG, whose gradient is O(1−p_o) near a saturated outcome (Proposition 3), ∇J_GracefulRL stays Θ(1) near fallback or error saturation. The lower bound rests entirely on Assumption 4: near pF→1, ‖gS‖≥δF, and near pE→1, inf_{λ∈[1−c,1]}‖λgS+(1−λ)gF‖≥δE. The only illustration is a three-action softmax with one logit per outcome (Appendix A.4), where the bounds follow immediately. For the actual ResNet and Qwen transformer policies, gS is a trajectory-level conditional expectation of sums of token score vectors, and the bound additionally requires the logit Jacobian not to annihilate these directions, as the authors state (b411). Nothing in the experimental sections measures these quantities. If gS→0 under fallback-saturated policies, or if the convex combination vanishes under error-saturated policies, GracefulRL's gradient also vanishes and the central theoretical distinction from standard policy gradient is not established. The authors acknowledge the proofs cover only population gradients (b372), but the unresolved gap is Assumption 4 itself, not the stochastic estimator.
The reader's weakest assumption is also the most load-bearing condition in the paper. The empirical demonstrations are suggestive, and the authors are appropriately candid about acknowledged limitations: online-only training, absent coding/agentic settings, and population-level proofs only (b210, b211, b212, b372). Those do not constitute objections. The unresolved issue is that Proposition 2/5's Θ(1) gradient guarantee — the stated reason GracefulRL avoids the collapse observed for GRPO/RLOO — depends on Assumption 4, which is neither proved for the trained architectures nor checked empirically. A saturated transformer policy could in principle have near-zero conditional score directions, in which case GracefulRL would stall exactly where the paper claims it cannot. This does not refute the empirical AIME or Minesweeper results, which could still hold as empirical findings, but it does mean the central mechanistic claim is conditional in a way the paper does not test. Because the reader's verdict is already CONDITIONAL and identifies the same assumption, my stress-test does not move the verdict; it sharpens the condition under which the paper should be reconsidered. The concrete test above would either verify Assumption 4 on real policies or demonstrate that the theoretical advantage is not realized, settling the concern directly.
The referee's report
If the results hold, the paper makes a useful dual contribution: it identifies a concrete optimization-dynamics failure mode of outcome-level RL with fallback rewards (gradient suppression near saturation) and provides a simple, analyzable surrogate that avoids it, with credible evidence across three quite different settings. The population-level proofs are algebraically correct under their stated assumptions (I checked the gradient identities in Appendix A.1, the O(1−p_o) bound in Proposition 3, and the Θ(1) argument in Proposition 5), and Appendix B's O(ϵ) bias analysis for the smoothing constant is sound. The empirical study is broad, the experimental design is mostly careful (validation-based checkpoint selection before test evaluation, several baselines, a held-out AbstentionBench), and the code is released. Two caveats are load-bearing: the non-degeneracy assumption underpinning the non-vanishing-gradient claim is unverified for the actual trained policies, and the LLM reliability frontiers come from prompt-embedded caution levels that make it hard to separate learned selective fallback from instruction adherence.
Pith's reading of the paper (a machine's summary, not the authors')
This paper tries to establish that reinforcement learning can shape not only how often a model succeeds, but also how it behaves when it is about to fail: a model can be trained to execute a safe fallback — abstaining, reverting a change, or handing off — whenever acting would be unreliable. The central obstacle is that the natural way to encode this preference, rewarding success above fallback above error and maximizing expected reward, is brittle: the policy-gradient signal shrinks to zero near a policy that has collapsed into always falling back or always erring, so training stalls. The paper's answer, GracefulRL, replaces the expected-reward objective with a log-barrier surrogate $(1-c)\log p_S + c\log(1-p_E)$ that keeps a non-vanishing gradient near those saturated policies. Because the behavior emerges from pure outcome-level rewards, no per-example abstention labels and no inference-time verifier or confidence threshold are needed. If correct, the work implies that outcome-level RL can teach a model to reason about the limits of its own ability and change course.
What carries the argument. The load-bearing object is the surrogate objective $$J_{GracefulRL} = (1-c)\log p_S + c\log(1-p_E),$$ a weighted sum of two log-probabilities defined on the ternary outcome simplex over Success, Fallback, and Error. It acts as a logarithmic barrier on the two undesirable boundaries — the error probability approaching one and the success probability approaching zero — so that under the preference ordering Success ≻ Fallback ≻ Error, moving probability mass away from a collapsed policy never loses optimization signal: the logarithmic form cancels the vanishing factor that suppresses the expected-reward gradient. The practical algorithm estimates this objective through inverse-frequency weights on sampled trajectories, with a small smoothing constant $\epsilon$ that trades bias for variance when successes or non-errors are rare.
Core claim. The central claim is that graceful failure — preserving success whenever possible while selectively executing a safe fallback when success is unlikely — can be learned end-to-end from outcome-level rewards alone, provided the optimization objective is chosen to survive the bad optimization regime. The paper proves that the expected-reward objective $J_{PG}=p_S + c p_F$ has gradients of order $O(1-p_o)$ as any outcome probability approaches one (Proposition 1), which explains why standard policy-gradient training collapses to a policy that always falls back and never solves the task. GracefulRL instead maximizes $J_{GracefulRL} = (1-c)\log p_S + c\log(1-p_E)$, whose gradient is $\Theta(1)$ near fallback or error saturation (Propositions 2 and 5), so the optimizer always retains a signal pointing away from the degenerate corners of the outcome simplex. Empirically, on the AIME benchmark, a Qwen3-4B-Base model trained this way reduces its error rate from 79.1% to 8.9% at caution level $c=0.8$, losing only 3.1 points of success relative to standard RLVR, and the same recipe improves or preserves both sides of the success-error tradeoff in maze navigation and multi-turn Minesweeper.
The premise it rests on. The load-bearing premise is the paper's non-degeneracy assumption (Assumption 4 in Appendix A.4): near a policy that is almost always falling back or almost always erring, the conditional score directions used by the gradient must stay bounded away from zero, and the trained network's parameterization must not annihilate them, with the paper illustrating this only on a three-action softmax toy and not verifying it on the ResNet or Qwen policies actually trained. The authors also acknowledge that the proofs cover population-level gradients only, so the practical claims assume the finite-rollout estimator with epsilon smoothing tracks the population objective.
If the paper is right (Pith's inferences, not the authors')
At test time the policy itself decides when to fall back, so no confidence threshold, verifier, or abstention labels are needed: the same trained model, prompted at its five caution levels, traces a reliability frontier that Pareto-dominates the calibration-reward baseline with test-time thresholding on AIME, AMC, JEEBench, and MATH500.
The headline number on AIME at $c=0.8$ is an error-rate reduction from 79.1% to 8.9%, an 88.8% relative drop, while success falls only 3.1 percentage points relative to binary-reward RL.
Fallback behavior transfers out of distribution: a model trained only on math problems raises its abstention F1 from 0.472 at $c=0.2$ to 0.677 at $c=0.8$ on AbstentionBench's unanswerable questions, above the 0.590 scored by an instruction-tuned model.
In sequential settings the fallback reward can act as reward shaping: in Minesweeper, GracefulRL at $c=0.5$ achieves 77.4% success and 6.0% error, beating standard MaxRL's 67.4% success and 17.0% error on both axes, and it concentrates errors away from the late-game turns where standard RL gambles.
The collapse of GRPO and RLOO into always-fallback or always-error policies is structural, traced to the $O(1-p_o)$ gradient suppression near saturation, so the paper argues that entropy bonuses and reward tuning cannot reliably fix it.
If the non-vanishing-gradient result survives real networks, the same ternary surrogate should transfer to other consequential action spaces the authors did not test, such as code agents that revert their own edits or database agents that roll back a transaction, since the formalism is agnostic to the action space.
The conditioning-suffix design trains several caution levels in one model, which suggests a testable extension: interpolating between the suffix prompts or their embeddings might yield intermediate caution levels the model never saw during training.
The gradient analysis is population-level, and the authors acknowledge this; a natural stress test is whether the finite-rollout estimator with epsilon smoothing preserves the constant-order gradient guarantee at small batch sizes, where rare successes make the inverse-frequency weights unstable.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Reinforcement learning makes language models better at solving tasks, but the same mechanism also rewards guessing. When success is the only way to earn reward, opting for a safe fallback is always worse than a long-shot attempt. When a wrong action can corrupt a database or ship broken code, that incentive can be costly. Therefore, we propose training models to fail gracefully: to preserve successful behavior whenever possible while learning to execute a safe fallback policy when acting would be unreliable. However, naively using policy gradients with intermediate reward for fallback behavior is brittle due to unfavorable optimization dynamics. We address this challenge with GracefulRL, which reshapes the optimization landscape to promote graceful failure. Importantly, this behavior emerges directly from outcome-level reward signals, without annotations specifying when to fall back or additional inference-time machinery such as a verifier. Across a range of tasks and scales, we train models that largely preserve their task-solving capability while selectively falling back when acting would be unreliable. More broadly, our results show that reinforcement learning can not only incentivize models to succeed, but also to reason about the limits of their capabilities and change course when they determine that success is out of reach.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-15
The authors' main claim, as the classifier picked it out 0.91
“More broadly, our results show that reinforcement learning can not only incentivize models to succeed, but also to reason about the limits of their capabilities and change course when they determine that success is out of reach.”
How the main claim is stated: asserted 0.99
What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.69 · human participants 0.04 · controlled comparison 0.14 · randomized assignment 0.01 · states quantitative result 0.04 · compares with prior results 0.22 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Reinforcement learning makes language models better at solving tasks, but the same mechanism also rewards guessing. | background 1.00 | 0.10 | 0.04 | asserted 1.00 |
| When success is the only way to earn reward, opting for a safe fallback is always worse than a long-shot attempt. | background 0.96 | 0.08 | 0.05 | asserted 1.00 |
| When a wrong action can corrupt a database or ship broken code, that incentive can be costly. | background 0.99 | 0.07 | 0.03 | asserted 0.72 |
| Therefore, we propose training models to fail gracefully: to preserve successful behavior whenever possible while learning to execute a safe fallback policy when acting would be unreliable. | contribution 0.98 | 0.28 | 0.33 | asserted 0.89 |
| However, naively using policy gradients with intermediate reward for fallback behavior is brittle due to unfavorable optimization dynamics. | gap 0.56 | 0.53 | 0.03 | asserted 1.00 |
| We address this challenge with GracefulRL, which reshapes the optimization landscape to promote graceful failure. | contribution 0.51 | 0.87 | 0.10 | asserted 1.00 |
| Importantly, this behavior emerges directly from outcome-level reward signals, without annotations specifying when to fall back or additional inference-time machinery such as a verifier. | method 0.55 | 0.88 | 0.12 | asserted 1.00 |
| Across a range of tasks and scales, we train models that largely preserve their task-solving capability while selectively falling back when acting would be unreliable. | result 0.99 | 0.96 | 0.28 | asserted 0.98 |
| More broadly, our results show that reinforcement learning can not only incentivize models to succeed, but also to reason about the limits of their capabilities and change course when they determine that success is out of reach. | interpretation 0.99 | 0.90 | 0.65 | asserted 0.99 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.92
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-15: 1 request, 6,809 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (19)
Figures are the authors', reproduced from the paper.
References (65)
- [1] Bengio, Yoshua and LeCun, Yann , booktitle =. Scaling Learning Algorithms Towards on Pith
- [2] Why Language Models Hallucinate , author=. 2025 , eprint= on Pith
- [3] Boyd, Stephen and Vandenberghe, Lieven , year=. Convex Optimization , publisher= on Pith
- [4] Hinton, Geoffrey E. and Osindero, Simon and Teh, Yee Whye , journal =. A Fast Learning Algorithm for Deep Belief Nets , volume = on Pith
- [5] Zhang, Hanning and Diao, Shizhe and Lin, Yong and Fung, Yi and Lian, Qing and Wang, Xingyao and Chen, Yangyi and Ji, Heng and Zhang, Tong. R -Tuning: Instructing Large Language Models to Say ` I Don ' t Know'. Proceedings of the 2024 Conference of the North American Chapter of... on Pithdoi:10.18653/v1/2024.naacl-long.394
- [6] Delightful Gradients Accelerate Corner Escape , author=. 2026 , eprint= on Pith
- [7] He, Kaiming and Zhang, Xiangyu and Ren, Shaoqing and Sun, Jian , title =. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , month = on Pith
- [8] Deep learning , author=. 2016 , publisher= on Pith
- [9] OpenAI o1 System Card , author=. 2026 , eprint= on Pith
- [10] Guo, Daya and Yang, Dejian and Zhang, Haowei and Song, Junxiao and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Zhang, Ruoyu and Ma, Shirong et al , year=. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning , volume=. Nature , publisher=. doi:10.1038/s4... on Pithdoi:10.1038/s41586-025-09422-z
- [11] Tulu 3: Pushing Frontiers in Open Language Model Post-Training , author=. Second Conference on Language Modeling , year= on Pith
- [12] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models , author=. 2024 , eprint= on Pith
- [13] DeepCoder: A Fully Open-Source 14B Coder at O3-mini Level , author= on Pith
- [14] Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedback , author=. 2024 , eprint= on Pith
- [15] Let's Verify Step by Step , author =. The Twelfth International Conference on Learning Representations , year = on Pith
- [16] Arora, Daman and Singh, Himanshu and Mausam. Have LLM s Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. doi:10.18653/v1/2023.emnlp-main.468 on Pithdoi:10.18653/v1/2023.emnlp-main.468
- [17] Vanishing Gradients in Reinforcement Finetuning of Language Models , author=. The Twelfth International Conference on Learning Representations , year= on Pith
- [18] Chow, C. , journal=. On optimum recognition error and reject tradeoff , year= on Pith
- [19] Ran El-Yaniv and Yair Wiener , title =. Journal of Machine Learning Research , year = on Pith
- [20] Liu, Ziyin and Wang, Zhikang and Liang, Paul Pu and Salakhutdinov, Russ R and Morency, Louis-Philippe and Ueda, Masahito , booktitle =. Deep Gamblers: Learning to Abstain with Portfolio Theory , url = on Pith
show 45 more
- [21] Elias Stengel-Eskin and Peter Hase and Mohit Bansal , booktitle=. 2024 , url= on Pith
- [22] Jixuan Leng and Chengsong Huang and Banghua Zhu and Jiaxin Huang , booktitle=. Taming Overconfidence in. 2025 , url= on Pith
- [23] Teaching Models to Express Their Uncertainty in Words , author=. Transactions on Machine Learning Research , issn=. 2022 , url= on Pith
- [24] Tian, Katherine and Mitchell, Eric and Zhou, Allan and Sharma, Archit and Rafailov, Rafael and Yao, Huaxiu and Finn, Chelsea and Manning, Christopher. Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Fee... on Pithdoi:10.18653/v1/2023.emnlp-main.330
- [25] Potsawee Manakul and Adian Liusie and Mark Gales , booktitle=. SelfCheck. 2023 , url= on Pith
- [26] Miao Xiong and Zhiyuan Hu and Xinyang Lu and YIFEI LI and Jie Fu and Junxian He and Bryan Hooi , booktitle=. Can. 2024 , url= on Pith
- [27] Language Models (Mostly) Know What They Know , author=. 2022 , eprint= on Pith
- [28] Scalable Best-of-N Selection for Large Language Models via Self-Certainty , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year= on Pith
- [29] Javier Garc\'ia and Fernando Fern\'andez , title =. Journal of Machine Learning Research , year = on Pith
- [30] Constrained Policy Optimization , author =. Proceedings of the 34th International Conference on Machine Learning , pages =. 2017 , editor = on Pith
- [31] Geifman, Yonatan and El-Yaniv, Ran , booktitle =. 2019 , editor = on Pith
- [32] Softmax Policy Gradient Methods Can Take Exponential Time to Converge , author =. Proceedings of Thirty Fourth Conference on Learning Theory , pages =. 2021 , editor = on Pith
- [33] Alekh Agarwal and Sham M. Kakade and Jason D. Lee and Gaurav Mahajan , title =. Journal of Machine Learning Research , year = on Pith
- [34] Dan Hendrycks and Collin Burns and Saurav Kadavath and Akul Arora and Steven Basart and Eric Tang and Dawn Song and Jacob Steinhardt , booktitle=. Measuring Mathematical Problem Solving With the. 2021 , url= on Pith
- [35] Zhai, Haotian and Liang, Jingcheng and Kang, Dongyeop. Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RL. Findings of the A ssociation for C omputational L inguistics: ACL 2026. 2026. doi:10.18653/v1/2026.findings-acl.985 on Pithdoi:10.18653/v1/2026.findings-acl.985
- [36] Zhepei Wei and Xiao Yang and Kai Sun and Jiaqi Wang and Rulin Shao and Jingxiang Chen and Mohammad Kachuee and Teja Gollapudi and Yiwei Liao and Nicolas SCHEFFER and Rakesh Wanga and Anuj Kumar and Yu Meng and Wen-tau Yih and Xin Luna Dong , booktitle=. Truth. 2026 , url= on Pith
- [37] Understanding the Impact of Entropy on Policy Optimization , author =. Proceedings of the 36th International Conference on Machine Learning , pages =. 2019 , editor = on Pith
- [38] KARL: Knowledge-Aware Reasoning and Reinforcement Learning for Knowledge-Intensive Visual Grounding , author=. 2026 , eprint= on Pith
- [39] Hen Davidov and Nachshon Cohen and Oren Kalinsky and Yaron Fairstein and Guy Kushilevitz and Ram Yazdi and Patrick Rebeschini , booktitle=. Knowing When to Quit: A Principled Framework for Dynamic Abstention in. 2026 , url= on Pith
- [40] Jiayun Wu and Jiashuo Liu and Zhiyuan Zeng and Tianyang Zhan and Tianle Cai and Wenhao Huang , booktitle=. Mitigating. 2026 , url= on Pith
- [41] TIAR: Trajectory-Informed Advantage Reweighting for LLM Abstention Learning , author=. 2026 , eprint= on Pith
- [42] Scaling Open-Ended Reasoning to Predict the Future , author=. 2026 , url= on Pith
- [43] GRACE: Reinforcement Learning for Grounded Response and Abstention under Contextual Evidence , author=. 2026 , eprint= on Pith
- [44] Williams, Ronald J. , journal =. 1992 , pages =. doi:10.1007/BF00992696 , volume = on Pithdoi:10.1007/bf00992696
- [45] Qwen3 Technical Report , author=. 2025 , eprint= on Pith
- [46] Miles: Enterprise-Grade Reinforcement Learning for Large-Scale Model Post-Training , author =. 2026 , howpublished = on Pith
- [47] Training a Generally Curious Agent , author=. Forty-second International Conference on Machine Learning , year= on Pith
- [48] Delightful Policy Gradient , author=. Reinforcement Learning Journal , volume=. 2026 , url= on Pith
- [49] Mehul Damani and Isha Puri and Stewart Slocum and Idan Shenfeld and Leshem Choshen and Yoon Kim and Jacob Andreas , booktitle=. Beyond Binary Rewards: Training. 2026 , url= on Pith
- [50] Qiying Yu and Zheng Zhang and Ruofei Zhu and Yufeng Yuan and Xiaochen Zuo and Yu Yue and Weinan Dai and Tiantian Fan and Gaohong Liu and Lingjun Liu and Xin Liu and Haibin Lin and Zhiqi Lin and Bole Ma and Guangming Sheng and Yuxuan Tong and Chi Zhang and Mofan Zhang and Wang ... on PithPith's reviewarXiv
- [51] Polina Kirichenko and Mark Ibrahim and Kamalika Chaudhuri and Samuel J. Bell , year=. AbstentionBench: Reasoning. 2506.09038 , archivePrefix= on PithPith's reviewarXiv
- [52] Proximal Policy Optimization Algorithms , author=. 2017 , eprint= on Pith
- [53] Mei, Jincheng and Xiao, Chenjun and Dai, Bo and Li, Lihong and Szepesvari, Csaba and Schuurmans, Dale , booktitle =. Escaping the Gravitational Pull of Softmax , url = on Pith
- [54] Tail-Likelihood Reinforcement Learning , author=. 2026 , eprint= on Pith
- [55] Ahmadian, Arash and Cremer, Chris and Gall. Back to Basics: Revisiting REINFORCE -Style Optimization for Learning from Human Feedback in LLM s. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024. doi:10.18653/v... on Pithdoi:10.18653/v1/2024.acl-long.662
- [56] Maximum Likelihood Reinforcement Learning , author=. Forty-third International Conference on Machine Learning , year= on Pith
- [57] Peter L Bartlett and Michael I Jordan and Jon D McAuliffe , title =. Journal of the American Statistical Association , volume =. 2006 , publisher =. doi:10.1198/016214505000000907 , URL = on Pithdoi:10.1198/016214505000000907
- [58] Tong Zhang , title =. The Annals of Statistics , number =. 2004 , doi = on Pith
- [59] Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones , author=. 2021 , eprint= on Pith
- [60] Training Verifiers to Solve Math Word Problems , author =. 2021 , eprint = on Pith
- [61] Self-Consistency Improves Chain of Thought Reasoning in Language Models , author =. The Eleventh International Conference on Learning Representations , year = on Pith
- [62] HybridFlow: A Flexible and Efficient RLHF Framework , author =. 2024 , journal = on Pith
- [63] Yang, Yuqing and Chern, Ethan and Qiu, Xipeng and Neubig, Graham and Liu, Pengfei , booktitle =. Alignment for Honesty , url =. doi:10.52202/079017-2030 , editor = on Pithdoi:10.52202/079017-2030
- [64] Sheng, Guangming and Zhang, Chi and Ye, Zilingfeng and Wu, Xibin and Zhang, Wang and Zhang, Ru and Peng, Yanghua and Lin, Haibin and Wu, Chuan , title =. Proceedings of the Twentieth European Conference on Computer Systems , pages =. 2025 , isbn =. doi:10.1145/3689031.3696075 ... on Pithdoi:10.1145/3689031.3696075
- [65] Tinus , title =. 2007 , howpublished = on Pith
Cited by 1 Pith paper
Reviewed papers on Pith that cite this one. The full citation record
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment