Pith. sign in

REVIEW 2 major objections 5 minor 79 references

Training language models to optimize the risk-coverage curve yields better selective prediction than accuracy or calibration rewards.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 01:50 UTC pith:BTGJYTPO

load-bearing objection First clean attempt to put AURC inside GRPO for LLM alignment; gains over RLVR/RLCR look real on ID/OOD and risk-controlled MedQA, batch ranking is the only soft approximation. the 2 major comments →

arxiv 2607.03528 v1 pith:BTGJYTPO submitted 2026-07-03 cs.LG cs.AIcs.CL

Aligning Language Models with Selective Prediction

Gaoxiang Luo , Yifan Wu , Sinian Zhang , Aryan Deshwal , Ju Sun This is my paper
classification cs.LG cs.AIcs.CL
keywords selective predictionrisk-coverage curveAURCLLM alignmentreinforcement learningverbalized confidencegroup relative policy optimization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

When large language models make high-stakes decisions, the safer design is often to answer only when the model is likely correct and to abstain otherwise, handing the rest to a human. That selective-prediction goal is measured by the area under the risk-coverage curve: how quickly error falls as coverage shrinks. Existing post-training methods reward correctness or calibrated confidence, but those objectives do not automatically produce the right ranking of correct versus incorrect answers. This paper introduces Reinforcement Learning for Selection Reward (RLSR), which replaces those rewards with a lifted, batch-ranked form of the area-under-risk-coverage objective inside a group-relative policy-optimization loop. Across multi-hop QA and multi-step math training sets, and on both in-domain and out-of-domain benchmarks, the resulting models separate correct from incorrect predictions more cleanly and therefore achieve lower risk at every useful coverage level, including a controlled 75 percent accuracy regime on a medical exam dataset.

Core claim

Directly aligning a language model with a selection-aware reward based on the area under the risk-coverage curve produces substantially better risk-coverage trade-offs than either correctness-only or correctness-plus-calibration alignment, on both the training distribution and held-out domains.

What carries the argument

Reinforcement Learning for Selection Reward (RLSR): a lifted, signed form of the weighted AURC that rewards correct rollouts and penalizes incorrect ones according to their rank inside a pooled mini-batch of B prompts times G samples, then feeds those ranks into group-relative policy optimization.

Load-bearing premise

Ranking the pooled rollouts inside each training batch is a close enough stand-in for the true population ranking that defines the area under the risk-coverage curve.

What would settle it

Train the same base models with identical hyperparameters but systematically smaller effective batch sizes; if the risk-coverage curves on held-out sets collapse toward the calibration or correctness baselines once the batch ranking becomes too noisy, the surrogate is inadequate.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes Reinforcement Learning for Selection Reward (RLSR), an LLM post-training alignment method that directly optimizes selective-prediction (SP) performance via a lifted, batch-approximated AURC reward inside the GRPO framework. It argues that confidence calibration is neither necessary nor sufficient for SP, replaces the usual correctness or Brier-style rewards with a signed, rank-weighted reward R_RLSR = ±α̂_i derived from the weighted form of AURC, and shows that the resulting models achieve lower AURC and higher Acc@10/25/50 than BASE, RLVR, and RLCR on HotPotQA- and BigMath-aligned Qwen2.5-7B and Llama-3.1-8B models, both in-domain and out-of-domain, with an additional risk-controlled MedQA deployment study.

Significance. If the empirical gains hold, the work supplies a practical, first-of-its-kind alignment objective that targets the risk-coverage trade-off rather than accuracy or ECE alone. The multi-model, multi-domain evaluation (including a high-stakes MedQA threshold-selection experiment), the explicit CC-vs-SP distinction, the lifted reward that supplies two-sided signals and margin enforcement, and the ablations on batch size, confidence scorers, and SFT baselines constitute a solid empirical package. The method is immediately usable with existing GRPO pipelines and verbalized or logit-based confidence, so the contribution is both conceptual and deployable.

major comments (2)
  1. Sec. 2.2 and Alg. 1: the central technical claim is that ranking the B×G pooled rollouts inside each mini-batch yields a sufficiently faithful surrogate for the population ranking that defines AURC_w. The only supporting evidence is the brief ablation (B=48/32/16, G=32 fixed) showing ID AURC stable at 0.44/0.44/0.45 while OOD AURC degrades from 0.41 to 0.48. That shows graceful degradation, not that the stochastic gradient remains unbiased for the population objective. A short theoretical argument (e.g., concentration of batch ranks, or a controlled experiment that freezes ranks to a large fixed pool) would strengthen the claim that the observed SP gains are not partly a batch-ranking artifact.
  2. Sec. 5 Limitation and Sec. 4.2: the paper itself notes that holistic AURC optimization is less directly useful than risk-constrained coverage maximization for deployment. The MedQA experiment (target 75 % accuracy) is the most practically relevant result, yet the training objective never sees a risk constraint. Either a risk-constrained variant of the reward or a clearer statement of how practitioners should choose the operating point after RLSR training would make the deployment claim more complete.
minor comments (5)
  1. Fig. 1 caption and surrounding text: the left panel is helpful, but the claim that perfect CC can still violate perfect SP ordering would be clearer with an explicit numerical example of two samples whose calibrated confidences reverse the desired ranking.
  2. Eq. (2.14)–(2.15): the equivalence AURC_w_lift = 2 AURC_w − 1 is stated; a one-line remark that the constant shift does not affect the GRPO advantage (zero-mean within group) would remove any residual doubt about the sign flip.
  3. Table 1 vs. Tables 8–11: the main table reports averages; the per-dataset tables reveal that on a few OOD sets (e.g., CommonsenseQA, GPQA under HotPotQA training) RLSR is not uniformly best. A short discussion of when the ranking signal fails would be useful.
  4. Typo: “Abalation study” (Sec. 4.1) should be “Ablation study”; “vise versa” appears twice and should be “vice versa”.
  5. Sec. C.4.2: LoRA rank r=1 is unusually low; a one-sentence justification or pointer to the cited “LoRA Without Regret” note would help reproducibility.

Circularity Check

0 steps flagged

No significant circularity: lifted AURC is a transparent algebraic reformulation of a standard SP metric, batch ranking is an acknowledged approximation, and empirical claims rest on external benchmarks rather than self-referential fits.

full rationale

The paper's derivation chain is: (i) adopt the standard AURC / weighted-AURC characterization of Zhou et al. (external, non-overlapping authors); (ii) form the lifted objective AURC_lift = 2 AURC_w - 1 by a sign flip on the binary indicators, which is algebraically equivalent and does not redefine the target in terms of itself; (iii) define the per-sample reward R_RLSR from the batch ranks of that lifted weight; (iv) plug the reward into standard GRPO. None of these steps is self-definitional, none fits a free parameter that is later reported as a prediction, and none imports a uniqueness theorem or ansatz from the present authors. The batch-ranking surrogate is flagged by the paper as Challenge (2) and is stress-tested by an ablation, not smuggled in as exact. Empirical superiority is measured against external baselines (BASE, RLVR, RLCR) on held-out ID/OOD datasets. Self-citations are limited to ordinary experimental-pipeline reuse and do not close any load-bearing logical loop. Score 0 is therefore the correct honest finding.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

As an empirical ML methods paper the load-bearing content is algorithmic and experimental rather than axiomatic. Free parameters are the usual training hyper-parameters; axioms are standard RL and SP definitions plus the cited AURC reformulation; the only invented entities are the lifted reward and the batch-ranking procedure.

free parameters (4)
  • effective batch size B×G
    Fixed at 1536 (G=32) for main runs; ablation shows mild degradation at smaller B. Directly controls quality of the ranking approximation to population AURC.
  • learning rate and schedule
    1e-5 constant (HotPotQA) or 5e-5 linear (BigMath); chosen by authors, not derived.
  • LoRA rank r=1, α=32
    Parameter-efficient fine-tuning choice taken from 'LoRA Without Regret'; affects capacity of the policy update.
  • generation temperature T=0.7 (train), T=0 (eval)
    Controls diversity of rollouts used for ranking and reward; evaluation is deterministic.
axioms (4)
  • standard math AURC admits the weighted empirical-risk form of Zhou et al. (Eq. 2.13) whose weights depend only on rank
    Used as the starting point for the reward; cited and not re-proved.
  • domain assumption GRPO (group-relative advantage + clipped density ratio) is a valid policy-gradient estimator for the expected reward
    Taken from DeepSeekMath / PPO literature; the paper only substitutes a new reward.
  • domain assumption Verbalized confidence extracted from <confidence> tags is a usable ranking signal for selective prediction
    Inherited from RLCR and recent confidence-elicitation work; ablation with log-prob shows the framework is not tied to it.
  • ad hoc to paper Lifted AURC (signed indicators) shares the same global minimizers as ordinary AURC
    Shown algebraically (AURC_lift = 2 AURC - 1); the paper notes that over-parameterization may still produce different generalization.
invented entities (2)
  • Lifted AURC reward R_RLSR = ±α̂_i no independent evidence
    purpose: Supplies two-sided learning signals and margin pressure between correct and incorrect predictions
    Defined in Eq. (2.15); no independent existence outside the proposed algorithm.
  • Batch-pooled ranking for mini-batch AURC approximation no independent evidence
    purpose: Makes the population ranking metric compatible with stochastic GRPO updates
    Described in Fig. 4 and Alg. 1; correctness rests on the empirical claim that B×G is large enough.

pith-pipeline@v1.1.0-grok45 · 33714 in / 2922 out tokens · 30823 ms · 2026-07-12T01:50:35.935452+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) are increasingly deployed as critical decision-making components in high-stakes real-world AI systems, rendering LLM reliability a foremost practical concern. In this paper, we focus on enhancing LLM reliability through selective prediction (SP), a strategy that allows an LLM to only predict for inputs where it is likely to be correct (i.e., coverage) and hence reduce the error rate (i.e., risk) on that portion of inputs -- flagging the remaining inputs for future human discretion. In other words, SP improves LLM reliability by balancing the risk-coverage trade-off and enabling seamless human-AI collaboration. To integrate SP into LLMs, we focus on the LLM post-training alignment stage and propose to align LLMs with SP performance metrics, in contrast with existing LLM alignment methods that focus primarily on correctness or calibration metrics. Specifically, we propose a novel alignment framework, Reinforcement Learning for Selection Reward (RLSR), which targets the area under the risk-coverage curve (AURC) -- a popular SP performance metric -- as its alignment objective. RLSR achieves substantially better risk-coverage trade-off compared to multiple alignment baselines on both in-domain and out-of-domain tasks.

Figures

Figures reproduced from arXiv: 2607.03528 by Aryan Deshwal, Gaoxiang Luo, Ju Sun, Sinian Zhang, Yifan Wu.

Figure 1
Figure 1. Figure 1: (Left) Perfect confidence calibration does not translate to perfect selective prediction, and vise versa; (Right) Our RLSR aligns LLMs with selective prediction (SP) and substantially improves upon competing methods in terms of SP metrics: AURC and Risk@ k. shown in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: SP for LLMs in deployment. Selective prediction (SP). For a general input￾output space X ′×Y′ and an associated data distribu￾tion D′ on X ′ ×Y′ , SP augments a predictive model f : X ′ → Y′ with a binary selector g : X ′ → {0, 1}, so that the final output by the pair (f, g) is: (f, g)(x ′ ) =  f(x ′ ) if g(x ′ ) = 1, abstain if g(x ′ ) = 0 ∀x ′ ∈ X ′ . (2.8) The selector g typically takes the form gs,τ (… view at source ↗
Figure 3
Figure 3. Figure 3: (Left) Normalized confidence distributions of correct and incorrect predictions. Empirically, lifted AURC as a reward achieves larger separation (margin) between correct and incorrect predictions on both train and test sets (see Sec. C.7); (Right) Behavior of the rank-based weights αb. lifted AURC as a reward, compared to the plain AURC. Consistent with AURCw lift, we define the per-sample RLSR reward as R… view at source ↗
Figure 4
Figure 4. Figure 4: Illustration of implementation. We pool rollouts from B inputs, each generating G samples, into an effective batch of size B × G (e.g., B = 2, G = 3). Rewards are computed based on ranking over the effective batch. The sub￾sequent relative advantage computation is identical to standard GRPO within each input group. To confirm that this definition of the per￾sample reward makes intuitive sense, in [PITH_FU… view at source ↗
Figure 5
Figure 5. Figure 5: The risk-coverage curves of the Qwen2.5-7B model aligned on HotPotQA. The in-domain [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The risk-coverage curves of the Qwen2.5-7B model aligned on BigMath. The in-domain [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The risk-coverage curves of the three align [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: The dashed curve is the oracle under which coverage levels left of c = 1 − rfull (orange) accept all correct predictions first and rank incor￾rect predictions last (red). rfull means full-coverage risk. The purple shaded area visualizes the SP gap. Confidence calibration (CC) for LLMs. Overconfidence in LLMs trained with RL with human feedback (RLHF) is widely recog￾nized [10, 15]. So, CC has been a domina… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

79 extracted references · 11 linked inside Pith

  1. [1]

    Large language models in law: A survey,

    J. Lai, W. Gan, J. Wu, Z. Qi, and P. S. Yu, “Large language models in law: A survey,”AI Open, vol. 5, pp. 181–196, 2024. [Online]. Available: https: //www.sciencedirect.com/science/article/pii/S2666651024000172

    2024

  2. [2]

    Ai-trader: Benchmarking autonomous agents in real-time financial markets,

    T. Fan, Y . Yang, Y . Jiang, Y . Zhang, Y . Chen, and C. Huang, “Ai-trader: Benchmarking autonomous agents in real-time financial markets,” 2025. [Online]. Available: https://arxiv.org/abs/2512.10971

    arXiv 2025

  3. [3]

    Evaluating large language model workflows in clinical decision support for triage and referral and diagnosis,

    F. Gaber, M. Shaik, F. Allega, A. J. Bilecz, F. Busch, K. Goon, V . Franke, and A. Akalin, “Evaluating large language model workflows in clinical decision support for triage and referral and diagnosis,”npj Digital Medicine, vol. 8, no. 1, p. 263, May 2025. [Online]. Available: https://doi.org/10.1038/s41746-025-01684-1

    doi:10.1038/s41746-025-01684-1 2025

  4. [4]

    Evaluating large language models for accuracy incentivizes hallucinations,

    A. T. Kalai, O. Nachum, S. S. Vempala, and E. Zhang, “Evaluating large language models for accuracy incentivizes hallucinations,”Nature, Apr 2026. [Online]. Available: https://doi.org/10.1038/s41586-026-10549-w

    doi:10.1038/s41586-026-10549-w 2026

  5. [5]

    On calibration of modern neural networks,

    C. Guo, G. Pleiss, Y . Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” inInternational conference on machine learning. PMLR, 2017, pp. 1321–1330

    2017

  6. [6]

    Measuring calibration in deep learning

    J. Nixon, M. W. Dusenberry, L. Zhang, G. Jerfel, and D. Tran, “Measuring calibration in deep learning.” inCVPR workshops, vol. 2, 2019

    2019

  7. [7]

    Calibrated selective classification,

    A. Fisch, T. S. Jaakkola, and R. Barzilay, “Calibrated selective classification,”Transactions on Machine Learning Research, 2022

    2022

  8. [8]

    Selective classification under distribution shifts,

    H. Liang, L. Peng, and J. Sun, “Selective classification under distribution shifts,”Transactions on Machine Learning Research, 2024. [Online]. Available: https://openreview.net/forum?id= dmxMGW6J7N

    2024

  9. [9]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models,

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y . K. Li, Y . Wu, and D. Guo, “Deepseekmath: Pushing the limits of mathematical reasoning in open language models,”CoRR, vol. abs/2402.03300, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2402.03300

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.03300 2024

  10. [10]

    Beyond binary rewards: Training LMs to reason about their uncertainty,

    M. Damani, I. Puri, S. Slocum, I. Shenfeld, L. Choshen, Y . Kim, and J. Andreas, “Beyond binary rewards: Training LMs to reason about their uncertainty,” inThe Fourteenth International Conference on Learning Representations, 2026. [Online]. Available: https://openreview.net/forum?id=ASQ649zdHm

    2026

  11. [11]

    Deepseek-r1 incentivizes reasoning in llms through reinforcement learning,

    D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, X. Zhang, X. Yu, Y . Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, B. Wang, B. Wu, B. Feng, C. Lu, C. Zhao, C. Deng, C. Ruan, D. Dai, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Xu, H. Ding, H. Gao, H. Qu, H. Li, J. G...

    doi:10.1038/s41586-025-09422-z 2025

  12. [12]

    Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base LLMs,

    X. Wen, Z. Liu, S. Zheng, S. Ye, Z. Wu, Y . Wang, Z. Xu, X. Liang, J. Li, Z. Miao, J. Bian, and M. Yang, “Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base LLMs,” inThe Fourteenth International Conference on Learning Representations, 2026. [Online]. Available: https://openreview.net/forum?id=jGbRWwIidy

    2026

  13. [13]

    How do LLMs compute verbal confidence?

    D. Kumaran, A. Conmy, F. Barbero, S. Osindero, V . Patraucean, and P. Veli ˇckovi´c, “How do LLMs compute verbal confidence?” inForty-third International Conference on Machine Learning, 2026. [Online]. Available: https://openreview.net/forum?id=t7oQT3uX66

    2026

  14. [14]

    Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs,

    M. Xiong, Z. Hu, X. Lu, Y . LI, J. Fu, J. He, and B. Hooi, “Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs,” inThe Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum?id=gjeQKFxFpZ

    2024

  15. [15]

    Taming overconfidence in LLMs: Reward calibration in RLHF,

    J. Leng, C. Huang, B. Zhu, and J. Huang, “Taming overconfidence in LLMs: Reward calibration in RLHF,” inThe Thirteenth International Conference on Learning Representations,

  16. [16]

    Available: https://openreview.net/forum?id=l0tg0jzsdL

    [Online]. Available: https://openreview.net/forum?id=l0tg0jzsdL

  17. [17]

    A novel characterization of the population area under the risk coverage curve (AURC) and rates of finite sample estimators,

    H. Zhou, J. V . Landeghem, T. Popordanoska, and M. B. Blaschko, “A novel characterization of the population area under the risk coverage curve (AURC) and rates of finite sample estimators,” inForty-second International Conference on Machine Learning, 2025. [Online]. Available: https://openreview.net/forum?id=LBBUJkqkOM

    2025

  18. [18]

    A survey of confidence estimation and calibration in large language models,

    J. Geng, F. Cai, Y . Wang, H. Koeppl, P. Nakov, and I. Gurevych, “A survey of confidence estimation and calibration in large language models,” inProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), K. Duh, H. Gomez, and S. Bethard, Eds. Mexic...

    2024

  19. [19]

    A survey of uncertainty estimation methods on large language models,

    Z. Xia, J. Xu, Y . Zhang, and H. Liu, “A survey of uncertainty estimation methods on large language models,” inFindings of the Association for Computational Linguistics: ACL 2025, W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, Eds. Vienna, Austria: Association for Computational Linguistics, Jul. 2025, pp. 21 381–21 396. [Online]. Available: https://a...

    2025

  20. [20]

    What does it take to build a performant selective classifier?

    S. Rabanser and N. Papernot, “What does it take to build a performant selective classifier?” in Advances in Neural Information Processing Systems, 2025

    2025

  21. [21]

    Adaptation with self-evaluation to improve selective prediction in LLMs,

    J. Chen, J. Yoon, S. Ebrahimi, S. O. Arik, T. Pfister, and S. Jha, “Adaptation with self-evaluation to improve selective prediction in LLMs,” inThe 2023 Conference on Empirical Methods in Natural Language Processing, 2023. [Online]. Available: https://openreview.net/forum?id=haPIkA8aOk

    2023

  22. [22]

    Calibrating LLMs for selective prediction: Balancing coverage and risk,

    Y . Mao, T. Durand, N. Mehrasa, J. He, and M. Ester, “Calibrating LLMs for selective prediction: Balancing coverage and risk,” inSocially Responsible and Trustworthy Foundation Models at NeurIPS 2025, 2025. [Online]. Available: https://openreview.net/forum?id=ZVZGjtP5VB

    2025

  23. [23]

    Optimal strategies for reject option classifiers,

    V . Franc, D. Prusa, and V . V oracek, “Optimal strategies for reject option classifiers,”Journal of Machine Learning Research, vol. 24, no. 11, pp. 1–49, 2023. [Online]. Available: http://jmlr.org/papers/v24/21-0048.html

    2023

  24. [24]

    Qwen2.5 technical report,

    Qwen, :, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X. Ren, Y . Fan, Y . Su, Y . Zhang, Y . Wan, Y . Liu, Z. Cui, Z. Zhang, ...

    Pith/arXiv arXiv 2025

  25. [25]

    The llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, 11 A. Spataru, B. Roziere, B. Biron, B. Tang, B. Chern, C. Caucheteux, C. Nayak, C. Bi, C. ...

  26. [26]

    Available: https://arxiv.org/abs/2407.21783

    [Online]. Available: https://arxiv.org/abs/2407.21783

    Pith/arXiv arXiv

  27. [27]

    HotpotQA: A dataset for diverse, explainable multi-hop question answering,

    Z. Yang, P. Qi, S. Zhang, Y . Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning, “HotpotQA: A dataset for diverse, explainable multi-hop question answering,” inProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii, Eds. Brussels, Belgium: Association for Computationa...

    2018

  28. [28]

    Big-math: A large-scale, high-quality math dataset for reinforcement learning in language models,

    A. Albalak, D. Phung, N. Lile, R. Rafailov, K. Gandhi, L. Castricato, A. Singh, C. Blagden, V . Xiang, D. Mahan, and N. Haber, “Big-math: A large-scale, high-quality math dataset for reinforcement learning in language models,” 2025. [Online]. Available: https://arxiv.org/abs/2502.17387

    Pith/arXiv arXiv 2025

  29. [29]

    Measuring short-form factuality in large language models,

    J. Wei, N. Karina, H. W. Chung, Y . J. Jiao, S. Papay, A. Glaese, J. Schulman, and W. Fedus, “Measuring short-form factuality in large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2411.04368

    Pith/arXiv arXiv 2024

  30. [30]

    TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension,

    M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer, “TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension,” inProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), R. Barzilay and M.-Y . Kan, Eds. Vancouver, Canada: Association for Computational Linguistics, Jul. 20...

    2017

  31. [31]

    GPQA: A graduate-level google-proof q&a benchmark,

    D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y . Pang, J. Dirani, J. Michael, and S. R. Bowman, “GPQA: A graduate-level google-proof q&a benchmark,” inFirst Conference on Language Modeling, 2024. [Online]. Available: https://openreview.net/forum?id=Ti67584b98

    2024

  32. [32]

    Measuring mathematical problem solving with the MATH dataset,

    D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt, “Measuring mathematical problem solving with the MATH dataset,” inThirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. [Online]. Available: https://openreview.net/forum?id=7Bywt2mQsCe

    2021

  33. [33]

    Training verifiers to solve math word problems,

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, “Training verifiers to solve math word problems,” 2021. [Online]. Available: https://arxiv.org/abs/2110.14168

    Pith/arXiv arXiv 2021

  34. [34]

    CommonsenseQA: A question answering challenge targeting commonsense knowledge,

    A. Talmor, J. Herzig, N. Lourie, and J. Berant, “CommonsenseQA: A question answering challenge targeting commonsense knowledge,” inProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), J. Burstein, C. Doran, and T. Solorio, Eds. Minn...

    2019

  35. [35]

    Outcome-based reinforcement learning to predict the future,

    B. Turtel, D. Franklin, K. Skotheim, L. Hewitt, and P. Schoenegger, “Outcome-based reinforcement learning to predict the future,”Transactions on Machine Learning Research,

  36. [36]

    Available: https://openreview.net/forum?id=bbhdeL8EUX

    [Online]. Available: https://openreview.net/forum?id=bbhdeL8EUX

  37. [37]

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams,

    D. Jin, E. Pan, N. Oufattole, W.-H. Weng, H. Fang, and P. Szolovits, “What disease does this patient have? a large-scale open domain question answering dataset from medical exams,”Applied Sciences, vol. 11, no. 14, 2021. [Online]. Available: https://www.mdpi.com/2076-3417/11/14/6421

    2021

  38. [38]

    Calibrating verbalized probabilities for large language models,

    C. Wang, G. Szarvas, G. Balazs, P. Danchenko, and P. Ernst, “Calibrating verbalized probabilities for large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2410.06707

    Pith/arXiv arXiv 2024

  39. [39]

    Selectively answering ambiguous questions,

    J. Cole, M. Zhang, D. Gillick, J. Eisenschlos, B. Dhingra, and J. Eisenstein, “Selectively answering ambiguous questions,” inProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for Computational Linguistics, Dec. 2023, pp. 530–543. [Online]. Available: https:...

    2023

  40. [40]

    Post-abstention: Towards reliably re-attempting the abstained instances in QA,

    N. Varshney and C. Baral, “Post-abstention: Towards reliably re-attempting the abstained instances in QA,” inProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A. Rogers, J. Boyd-Graber, and N. Okazaki, Eds. Toronto, Canada: Association for Computational Linguistics, Jul. 2023, pp. 967–982. [On...

    2023

  41. [41]

    Conformal language modeling,

    V . Quach, A. Fisch, T. Schuster, A. Yala, J. H. Sohn, T. S. Jaakkola, and R. Barzilay, “Conformal language modeling,” inThe Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum?id=pzUhfQ74c5

    2024

  42. [42]

    Controllable text generation for large language models: A survey,

    X. Liang, H. Wang, Y . Wang, S. Song, J. Yang, S. Niu, J. Hu, D. Liu, S. Yao, F. Xiong et al., “Controllable text generation for large language models: A survey,”arXiv preprint arXiv:2408.12599, 2024

    Pith/arXiv arXiv 2024

  43. [43]

    Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback,

    K. Tian, E. Mitchell, A. Zhou, A. Sharma, R. Rafailov, H. Yao, C. Finn, and C. Manning, “Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback,” inProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali, Eds. Singa...

    2023

  44. [44]

    Semantic calibration of LLMs through the lens of temperature scaling,

    T. A. Lamb, D. R. Ivanova, P. Torr, and T. G. J. Rudner, “Semantic calibration of LLMs through the lens of temperature scaling,” inICLR Workshop: Quantify Uncertainty and Hallucination in Foundation Models: The Next Frontier in Reliable AI, 2025. [Online]. Available: https://openreview.net/forum?id=22UNXHzBgt

    2025

  45. [45]

    Calibrating the confidence of large language models by eliciting fidelity,

    M. Zhang, M. Huang, R. Shi, L. Guo, C. Peng, P. Yan, Y . Zhou, and X. Qiu, “Calibrating the confidence of large language models by eliciting fidelity,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y . Al-Onaizan, M. Bansal, and Y .-N. Chen, Eds. Miami, Florida, USA: Association for Computational Linguistics, No...

    2024

  46. [46]

    Calibrating large language models with sample consistency,

    Q. Lyu, K. Shridhar, C. Malaviya, L. Zhang, Y . Elazar, N. Tandon, M. Apidianaki, M. Sachan, and C. Callison-Burch, “Calibrating large language models with sample consistency,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 18, pp. 19 260–19 268, Apr. 2025. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/34120

    2025

  47. [47]

    Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model,

    J. Hu, Y . Zhang, Q. Han, D. Jiang, X. Zhang, and H.-Y . Shum, “Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model,” inThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. [Online]. Available: https://openreview.net/forum?id=NFM8F5cV0V

    2026

  48. [48]

    Bnpo: Beta normalization policy optimization,

    C. Xiao, M. Zhang, and Y . Cao, “Bnpo: Beta normalization policy optimization,” 2025. [Online]. Available: https://arxiv.org/abs/2506.02864

    Pith/arXiv arXiv 2025

  49. [49]

    LoRA: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” inInternational Conference on Learning Representations, 2022. [Online]. Available: https://openreview.net/forum?id=nZeVKeeFYf9

    2022

  50. [50]

    Lora without regret,

    J. Schulman and T. M. Lab, “Lora without regret,”Thinking Machines Lab: Connectionism, 2025, https://thinkingmachines.ai/blog/lora/

    2025

  51. [51]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” inProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

    2023

  52. [52]

    FlashAttention-2: Faster attention with better parallelism and work partitioning,

    T. Dao, “FlashAttention-2: Faster attention with better parallelism and work partitioning,” in International Conference on Learning Representations (ICLR), 2024

    2024

  53. [53]

    Capabilities of gemini models in medicine,

    K. Saab, T. Tu, W.-H. Weng, R. Tanno, D. Stutz, E. Wulczyn, F. Zhang, T. Strother, C. Park, E. Vedadi, J. Z. Chaves, S.-Y . Hu, M. Schaekermann, A. Kamath, Y . Cheng, D. G. T. Barrett, C. Cheung, B. Mustafa, A. Palepu, D. McDuff, L. Hou, T. Golany, L. Liu, J. baptiste Alayrac, N. Houlsby, N. Tomasev, J. Freyberg, C. Lau, J. Kemp, J. Lai, S. Azizi, K. Kana...

    Pith/arXiv arXiv 2024

  54. [54]

    Medgemma technical report,

    A. Sellergren, S. Kazemzadeh, T. Jaroensri, A. Kiraly, M. Traverse, T. Kohlberger, S. Xu, F. Jamil, C. Hughes, C. Lau, J. Chen, F. Mahvar, L. Yatziv, T. Chen, B. Sterling, S. A. Baby, S. M. Baby, J. Lai, S. Schmidgall, L. Yang, K. Chen, P. Bjornsson, S. Reddy, R. Brush, K. Philbrick, M. Asiedu, I. Mezerreg, H. Hu, H. Yang, R. Tiwari, S. Jansen, P. Singh, ...

    Pith/arXiv arXiv 2025

  55. [55]

    Your task is to point out things where the model could be wrong in its thinking, or things where there might be ambiguity in the solution steps, or in the reasoning process itself

  56. [56]

    You should not suggest ways of fixing the response, your job is only to reason about uncertainties

  57. [57]

    In these cases, it is also okay to have only a small number of uncertainties and then explicitly say that I am unable to spot more uncertainties

    For some questions, the response might be correct. In these cases, it is also okay to have only a small number of uncertainties and then explicitly say that I am unable to spot more uncertainties. 16

  58. [58]

    For example, uncertainties may arise from ambiguities in the question, or from the application of a particular lemma/proof

    Uncertainties might be different from errors. For example, uncertainties may arise from ambiguities in the question, or from the application of a particular lemma/proof

  59. [59]

    If there are alternate potential approaches that may lead to different answers, you should mention them

  60. [60]

    List out plausible uncertainties, do not make generic statements, be as specific about uncertainties as possible

  61. [61]

    Enclose this uncertainty analysis within<analysis></analysis>tags. The final format that must be followed is: <think> reasoning process here </think><answer> final answer here </answer><analysis> analysis about confidence and uncertainty here </analysis><confidence> confidence level here (number between 0 and 1)</confidence> C.2 Verifiers Verifiers are us...

  62. [62]

    This verifier is used for HotPotQA and HotPotQA-Modified

    Exact-Match.The predicted answer must exactly match the ground truth answer string. This verifier is used for HotPotQA and HotPotQA-Modified

  63. [63]

    This verifier is used for Math-500, GSM8K, and Big-Math Digits

    Math-Verify.We use math-verify, a robust mathematical expression evaluation system that checks semantic equivalence of mathematical expressions. This verifier is used for Math-500, GSM8K, and Big-Math Digits

  64. [64]

    YES” or “NO

    LLM-as-a-Judge.We use Llama-3.1-8B-Instruct with greedy decoding (temperature=0) as the judge model. The judge is provided with the question, the groundtruth answer, and the predicted answer, and is prompted to respond with “YES” or “NO” based on correctness. Importantly, we do not condition the judge on thinking traces to avoid potential biases. This ver...

  65. [65]

    This modified version systematically varies the availability of supporting evidence by removing zero, one, or both of the key para- graphs required to answer each question

    HotPotQA-Modified.We adopt the HotpotQA-Modified dataset introduced by [10], which is derived from the original HotPotQA distractor dataset [25]. This modified version systematically varies the availability of supporting evidence by removing zero, one, or both of the key para- graphs required to answer each question. The questions are evenly distributed a...

  66. [66]

    BigMath.We use BigMath [26], a large-scale curated dataset for reinforcement learning that contains over 250,000 mathematical problems. Following [10], we retain only problems with LLaMA-8B solve rates between 0% and 70% to maintain appropriate difficulty, and further restrict to problems with numerical answers to enable reliable automatic verification—re...

  67. [67]

    Thus, each question contains 8 paragraphs with both supporting paragraphs present

    HotPotQA.(CC BY-SA 4.0 License) We use 1,000 validation samples from the original Hot- PotQA distractor dataset [25], with 2 irrelevant paragraphs removed from each question. Thus, each question contains 8 paragraphs with both supporting paragraphs present. Correctness is measured using exact-match

  68. [68]

    Correctness is measured using exact-match

    HotPotQA-Modified.(MIT License) We use 500 held-out validation samples from the modified dataset introduced by [10]. Correctness is measured using exact-match

  69. [69]

    Correctness is measured usingmath-verify

    Math-500.(MIT License) We use the MATH-500 dataset, a subset of problems from the original MATH dataset [30]. Correctness is measured usingmath-verify. 18

  70. [70]

    Correctness is measured usingmath-verify

    GSM8K.(MIT License) We use 1,319 problems from the test set of the Grade School Math 8K dataset [31]. Correctness is measured usingmath-verify

  71. [71]

    Correctness is measured usingmath-verify

    Big-Math Digits.(MIT License) We use 1,000 held-out validation samples from the filtered Big-Math dataset [26]. Correctness is measured usingmath-verify

  72. [72]

    Correctness is measured using LLM-as-a-judge

    TriviaQA.(Apache 2.0 License) We use 2,000 samples from the TriviaQA [28] validation set, specifically the no-context split, to test the factual accuracy. Correctness is measured using LLM-as-a-judge

  73. [73]

    Correctness is measured using LLM-as-a-judge

    SimpleQA.(MIT License) We use the full dataset consisting of 4,326 factual questions [27]. Correctness is measured using LLM-as-a-judge

  74. [74]

    Correctness is measured using LLM-as-a-judge

    CommonsenseQA.(MIT License) uses 1,220 problems from the CommonsenseQA [32] vali- dation set, a multiple-choice question answer dataset requiring various types of commonsense knowledge. Correctness is measured using LLM-as-a-judge

  75. [75]

    Correctness is measured using LLM-as-a-judge

    GPQA.(MIT License) We use the main dataset (gpqa_main) that contains 448 multiple-choice questions written by experts in the fields of biology, physics, and chemistry [29]. Correctness is measured using LLM-as-a-judge

  76. [76]

    Correctness is measured using LLM-as-a-judge

    MedQA.(MIT License) We use the 4-option version of MedQA [34], consistent with its original assessment and usage by technical reports of frontier medical LLMs [50, 51]. Correctness is measured using LLM-as-a-judge. C.5.2 Evaluation Metrics

  77. [77]

    We refer the reader to Sec

    AURC (↓).Thearea under risk-coverage curvemeasures cumulative selective risk as a function of coverage with sorted prediction confidence. We refer the reader to Sec. 2.1 for the formal definition. Lower values means better SP performance

  78. [78]

    We useM= 10bins

    ECE (↓).Theexpected calibration errormeasures the alignment between confidence and correct- ness likelihood: ECE= XM m=1 |Bm|/N· |acc(B m)−conf(B m)|(C.2) where M is the number of bins, Bm is the set of samples in bin m, and N is the total number of samples. We useM= 10bins

  79. [79]

    D. Nitrofurantoin

    Accuracy@k% (↑).Accuracy computed on the top k% most confident predictions, measuring how well the model’s confidence identifies its correct predictions. We report results at coverage levelsk∈ {10,25,50}. C.6 Mimicking Selective Prediction in Deployment C.6.1 The MedQA Dataset We use the 4-option version of MedQA [34], consistent with its original assessm...