Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Reinforcement Learning from Human Feedback with High-Confidence Safety Constraints

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read HC-RLHF trains language models under a deliberately pessimistic safety constraint and only returns a model if a held-out statistical test certifies that its expected harm is below tolerance with probability at least 1−δ.

desk verdict Seldonian-style safety guarantee for RLHF that is close, but the proof has a real gap: the safety test must sample responses from the candidate policy, and Algorithm 1 never says it does. read the letter →

arxiv 2506.08266 v1 pith:E3VDQ5S7 submitted 2025-06-09 cs.LG cs.AIcs.CLstat.AP

Yaswanth Chittepu , Blossom Metevier , Will Schwarzer , Austin Hoag , Scott Niekum , Philip S. Thomas This is my paper
classification cs.LGcs.AIcs.CLstat.AP
keywords reinforcementlearningfromhumanfeedbackLLMalignmentsafetyconstraintshigh-confidenceguaranteescostmodelharmlessnessStudent'st-testtest
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes High-Confidence RLHF (HC-RLHF), a way to align language models with human preferences while providing a statistical safety certificate. It separates helpfulness from harmlessness by training separate reward and cost models, then runs a two-step pipeline: first it optimizes under an intentionally pessimistic safety constraint, then it tests the resulting policy on held-out data. The algorithm returns the policy only when a Student's t upper-confidence bound on expected harm is at or below a user-chosen tolerance; otherwise it returns a no-solution outcome that is treated as safe. The central result is a bound on the probability of ever returning an unsafe policy, and experiments on three language models show fewer harmful responses and improved helpfulness compared with an existing constrained-RLHF baseline.

What carries the argument

The load-bearing object is the high-confidence upper bound on expected cost, U_ttest(ĝ) = sample mean + (sample standard deviation / √m) · t_{1−δ,m−1}, built from Student's t quantiles. This bound appears twice: first in candidate selection, where the training constraint inflates the empirical mean cost by a standard-deviation term K(δ) to make the optimization pessimistic about safety, and second in the held-out safety test that decides whether to return the candidate or to return no solution. The policy-gradient update implementing this pessimistic constraint uses an augmented reward containing the empirical mean and standard deviation of costs, with RLOO variance reduction.

What would settle it

Run HC-RLHF with a cost model whose harm scores are heavily skewed and bounded, fixing δ = 0.1 and a small safety-test size such as m = 50, then repeatedly measure the true expected cost of returned policies; if the empirical failure rate clearly exceeds 10%, the advertised 1−δ guarantee does not hold outside the normality assumption.

Watch

Extended reading notes

Core claim

The paper's central claim is Theorem 4.2: for Algorithm 1, with g defined as the expected cost under the learned cost model minus a tolerance τ, and under the assumption that the mean of the safety-test estimates is normally distributed, Pr(g(alg(D)) ≤ 0) ≥ 1−δ. In other words, the algorithm either returns a candidate policy whose expected harm is certified by a Student's t upper-confidence bound to be at most τ with confidence at least 1−δ, or it returns a no-solution outcome whose harm is defined as zero. The proof shows that the safety-test estimates are unbiased and i.i.d., making the Student's t bound valid, and that the algorithm returns a policy only when that bound is non-positive.

Load-bearing premise

The certificate is exact only if the safety-test cost estimates are unbiased i.i.d. draws from the candidate policy's response distribution and their sample mean is normally distributed, so that Student's t-test gives the claimed confidence.

Editorial extensions

If this is right

  • If the theorem holds, a deployed model is either certified by the safety test to respect the harm tolerance with confidence at least 1−δ, or the algorithm returns no solution and no unsafe model is released.
  • Because the final decision is made on held-out data by the safety test, the method cannot silently return an over-optimized policy that only looks safe under the training objective.
  • The same safety-test machinery can use distribution-free bounds such as Hoeffding's inequality, extending the high-confidence guarantee to settings where harm scores are known to be bounded.
  • Experiments indicate that HC-RLHF stays safe across stricter tolerance settings (more negative τ), whereas the constrained baseline without the safety test fails at stricter thresholds.
  • Users can choose the confidence level δ and the harm tolerance τ ahead of time, so the safety guarantee is tuned to the risk profile of the application.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The certificate applies to expected cost under the learned cost model, not directly to real-world harm; if the cost model mislabels harmful content, the guarantee does not cover that error.
  • The exact 1−δ claim depends on the sample mean of safety-test estimates being normally distributed; with heavily skewed or heavy-tailed harm scores and small safety sets, the Student's t bound is only an approximation and the stated confidence may be optimistic.
  • Treating the no-solution outcome as safe shifts the practical burden: an algorithm that frequently returns no solution is formally safe but may be unusable, so the data size and threshold needed to avoid frequent no-solution outcomes become important operational questions.
  • The guarantee assumes a stationary prompt distribution between training and deployment; adversarial rephrasing or shifts in prompt frequency are outside the certified claim, as the paper notes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes HC-RLHF, a Seldonian-style algorithm for RLHF that decouples helpfulness and harmlessness by training separate reward and cost models, then adds a held-out safety test: after candidate selection under a pessimistic cost constraint, the algorithm computes a Student-t high-confidence upper bound on expected cost and returns the candidate only if that bound is at or below zero, otherwise returning NSF. The main theoretical claim (Theorem 4.2) is that, under Assumption 4.1, the probability that HC-RLHF returns an unsafe policy is at most delta. The experiments compare HC-RLHF with Safe RLHF on Qwen2-1.5B, Qwen2.5-3B, and Llama3.2-3B, reporting win rates, scatter plots of reward versus cost, a 30-trial failure-rate study for one base model, and a single-trial threshold sweep.

Significance. If the guarantee can be stated and proved correctly, the paper would make a meaningful contribution: it is the first RLHF method, to my knowledge, that attempts to provide a Seldonian high-confidence safety guarantee on expected harm while maximizing helpfulness. The Seldonian meta-architecture is standard, the decision to keep the safety test independent of candidate selection is principled, and the code is publicly available. The paper also correctly identifies that the candidate-selection penalty need not be valid for the final safety guarantee to hold, since the safety test uses the held-out split. However, the central theorem is not valid as written because the safety-test estimator is not shown to be on-policy and the threshold tau is omitted from the test statistic; these issues are load-bearing and must be repaired before the contribution can be assessed.

major comments (3)
  1. [Algorithm 1, line 4; Section 4 proof] The unbiasedness claim in the proof of Theorem 4.2 is unsupported as written. The theorem requires E[g_hat_i] = g(theta_c) = E_{x~D_x, y~pi_{theta_c}(.|x)}[C_psi(x,y)] - tau, but Algorithm 1 partitions a fixed input dataset into D_c and D_s and sets g_hat_i = C_psi(x_i,y_i) for the stored pairs. The paper never states that the response y_i is sampled from the candidate policy pi_{theta_c} for the safety test, nor does it provide an importance-weighting or other off-policy correction. If D_s was generated by the SFT policy, the preference dataset, or any policy other than theta_c, then E[g_hat_i] is the expected cost under that behavior policy, not under the candidate policy, and the t-test bounds the wrong quantity. Please specify the generative process for D_s (e.g., draw prompts from a held-out prompt pool and sample responses from the candidate after candidate selection) and adjust the algorithm and proof accordingly.
  2. [Algorithm 1, line 4; Eq. (8)] Even if the responses in D_s were on-policy, the estimator in line 4 is g_hat_i = C_psi(x_i,y_i), whose expectation is E[C_psi], whereas the performance function g in Eq. (8) is E[C_psi] - tau. The safety test U_ttest(g_hat) <= 0 therefore certifies E[C_psi] <= 0, not E[C_psi] - tau <= 0. For the negative thresholds tau in Table 2, this is a strictly weaker condition than g(theta_c) <= 0, so passing the test does not imply the policy is safe according to the paper's own definition. The proof's equality E[g_hat_i] = g(theta_c) is inconsistent with the definition in line 4. The fix is to define g_hat_i = C_psi(x_i,y_i) - tau and test U_ttest(g_hat) <= 0, or to restrict the theorem to tau = 0.
  3. [Abstract; Section 3.1; Assumption 4.1] The abstract advertises an unconditional probabilistic guarantee: 'proof that it will not return an unsafe solution with a probability greater than a user-specified threshold.' This is stronger than what the paper proves. Theorem 4.2 depends on Assumption 4.1, which requires the estimates g_hat to be normally distributed (or at least the sample mean to be normal), and Section 3.1 explicitly concedes that Student's t-test is exact only under normality and relies on the CLT as a large-m approximation. For the finite sample sizes and non-normal cost-model outputs used in the experiments, the advertised 1-delta confidence is not exact. The abstract should either state the normality assumption or the paper should use the Hoeffding bound from Appendix B with a stated boundedness assumption on C_psi. This is a central claim and should be reported accurately.
minor comments (5)
  1. [Section 2.3; Algorithm 1] Section 2.3 says that the training dataset D 'consists of prompts sampled from D_x', but Algorithm 1 iterates over (x_i,y_i) pairs in D_s. Please clarify what D contains and where the responses y_i come from; this is part of the under-specification of the safety-test sampling.
  2. [Section 5.1, Seldonian Guarantee] The statement that the failure rate 'was observed to be 0, with a standard deviation of 0' is misleading. With 30 trials and zero failures, the standard deviation of the observed proportion is not the right uncertainty measure; a binomial confidence interval (e.g., approximate 95% CI of [0, 0.116]) would be more informative. Also, this experiment uses only Qwen2-1.5b, and the Llama3.2-3b threshold sweep in Table 2 is a single trial, so the empirical support for the probabilistic guarantee across models is currently thin.
  3. [Table 2] Table 2 reports one trial per threshold. A single trial cannot demonstrate a 1-delta property, and the table should be explicitly labeled as an illustrative run rather than as validation of the Seldonian guarantee.
  4. [Appendix A] There is a typo in the REINFORCE review: 'cummulative rewrad' should be 'cumulative reward'.
  5. [Section 3.2, K(delta)] The candidate-selection penalty K(delta) includes free coefficients rho_1 and rho_2, but the paper does not state explicitly that these coefficients affect only the candidate-selection stage and do not enter the validity of the final safety test. Since the safety test uses only D_s, this is not a threat to the theorem, but stating it would prevent confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the high-confidence safety guarantee is produced by a held-out Student's t-test bound on independently defined estimates, not by reusing fitted values or by a self-citation chain.

full rationale

Walking the derivation chain: Algorithm 1 selects a candidate using D_c, then computes ghat_i = C_psi(x_i,y_i) on the held-out split D_s (line 4) and returns the candidate only when U_ttest(ghat) <= 0 (line 5). Theorem 4.2's proof uses only the safety-test step: if ghat is i.i.d. and unbiased for g(theta_c), Student's t-test (an external result, cited to Student 1908) supplies Pr(g(theta_c) > U_ttest(ghat)) <= delta, and the decision rule makes Pr(g(alg(D)) > 0) <= delta; NSF is safe by the paper's explicit convention g(NSF) = 0. The pessimistic candidate-selection constraint is not used in the guarantee, so it is not a fitted input disguised as a prediction. The cost model C_psi defines the safety objective rather than being trained to satisfy the final bound, so its use is a modeling choice, not circularity. Self-citations (Thomas et al. 2019, Metevier et al. 2019, Weber et al. 2022) describe the Seldonian meta-architecture the paper instantiates, but the theorem is proved in-paper from Student's t-test and does not reduce to those citations. The proof does contain a rigor gap: it asserts unbiasedness of ghat_i without specifying that D_s responses are generated from the returned candidate policy theta_c (or an off-policy correction), which is a correctness concern, not a circularity. No equation or fitted parameter is shown to be equivalent to the claimed result by construction, so the circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central guarantee rests on the four axioms above; the normality assumption is the most fragile because the paper relies on it for the exact t-test bound. The candidate-selection penalty contains two hand-tuned constants that influence performance but not the validity of the safety test. No invented entities are introduced.

free parameters (2)
  • rho_1 = 4
    Hand-tuned scaling coefficient in the candidate-selection confidence penalty K(delta), chosen empirically to balance safety and helpfulness in Section 3.2; it affects how often a candidate passes, not the validity of the final t-test.
  • rho_2 = 2
    Second hand-tuned scaling coefficient in the same K(delta) expression; like rho_1, it is selected by empirical tuning rather than derived from the confidence bound.
assumptions (4)
  • domain assumption The sample mean of the m safety-test estimates is normally distributed (Assumption 4.1).
    Theorem 4.2 uses this to apply Student's t-test exactly; the paper concedes normality only holds approximately via the CLT, so the finite-sample 1-delta guarantee is conditional on this premise.
  • domain assumption The prompt distribution is stationary between training and deployment.
    Stated at the end of Section 4; distribution shift can degrade the safety guarantee because the expectation in g is taken under the training-time prompt distribution.
  • domain assumption The learned cost model C_psi is the operative definition of harm for the guarantee.
    The performance function g is defined as expected C_psi minus tau, so the 1-delta guarantee is relative to the cost model, not to real-world harm; the abstract's word unsafe is stronger than what is proved.
  • domain assumption Returning no solution (NSF) is safe, i.e., g(NSF)=0.
    This Seldonian convention is used in the proof of Theorem 4.2; if a practitioner falls back to a base model, that fallback model is not covered by the guarantee.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reinforcement Learning from Human Feedback with High-Confidence Safety Constraints." pith.science (2026). https://pith.science/paper/E3VDQ5S7

@misc{pith2026250608266,
  author       = {Pith},
  title        = {Pith review of: Reinforcement Learning from Human Feedback with High-Confidence Safety Constraints},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E3VDQ5S7}},
  note         = {Machine review of arXiv:2506.08266}
}
read the original abstract

Existing approaches to language model alignment often treat safety as a tradeoff against helpfulness, which can lead to unacceptable responses in sensitive domains. To ensure reliable performance in such settings, we propose High-Confidence Safe Reinforcement Learning from Human Feedback (HC-RLHF), a method that provides high-confidence safety guarantees while maximizing helpfulness. Similar to previous methods, HC-RLHF explicitly decouples human preferences into helpfulness and harmlessness (safety), which are learned by training a reward model and a cost model, respectively. It then employs a two-step process to find safe solutions. In the first step, it optimizes the reward function under an intentionally pessimistic version of the cost constraint. In the second step, the trained model undergoes a safety test to verify whether its performance stays within an upper-confidence bound of the actual cost constraint. We provide a theoretical analysis of HC-RLHF, including proof that it will not return an unsafe solution with a probability greater than a user-specified threshold. For our empirical analysis, we apply HC-RLHF to align three different language models (Qwen2-1.5B, Qwen2.5-3B, and LLaMa3.2-3B) with human preferences. Our results demonstrate that HC-RLHF produces safe models with high probability and can improve harmlessness and helpfulness compared to previous methods.

Figures

Figures reproduced from arXiv: 2506.08266 by the authors.

Figure 1
Figure 1. A common Seldonian meta-architecture: Given training data [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Scatter plots of reward vs. cost on the test set for different training methods. The top row corresponds to [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Win rate and safety distribution visualizations for LLaMA3.2-3B and Qwen2.5-3B, evaluated using the [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Breakdown of HC-RLHF win, tie, and lose rates vs. Safe-RLHF across different safety-related categories in [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Scatter plots for the rewards vs costs on the test split of the data for the Qwen2-1.5b model. Points to the [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Visualizations of win rates and safety distributions for Qwen2-1.5b, evaluated using our trained reward and [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Adaptive Margin RLHF via Preference over Preferences

    cs.LG 2025-09 conditional novelty 6.0 of 10

    Adaptive margins for DPO inferred from preference-over-preference comparisons improve alignment quality, with random sampling of comparisons working best overall.

Reference graph

Works this paper leans on

58 extracted references · 22 canonical work pages · cited by 1 Pith paper

  1. [1]

    Back to basics: Revisiting reinforce style optimization for learning from human feedback in LLMs , 2024

    Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in LLMs , 2024. URL https://arxiv.org/abs/2402.14740

    arXiv 2024

  2. [2]

    Constrained Markov decision processes

    Eitan Altman. Constrained Markov decision processes. Routledge, 2021

    2021

  3. [3]

    Constitutional AI : Harmlessness from AI feedback

    Yuntao Bai et al. Constitutional AI : Harmlessness from AI feedback. ArXiv, abs/2212.08073, 2022 a . URL https://api.semanticscholar.org/CorpusID:254823489

    arXiv 2022

  4. [4]

    Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022 b

    Yuntao Bai et al. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022 b . URL https://arxiv.org/abs/2204.05862

    arXiv 2022

  5. [5]

    Convex optimization

    Stephen P Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004

    2004

  6. [6]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. T he method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952

    work page 1952

  7. [7]

    Brown, Miljan Martic, Shane Legg, and Dario Amodei

    Paul Francis Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. ArXiv, abs/1706.03741, 2017. URL https://api.semanticscholar.org/CorpusID:4787508

    arXiv 2017

  8. [8]

    Safe RLHF : Safe reinforcement learning from human feedback

    Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF : Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310.12773, 2023

    arXiv 2023

Show all 58 references
  1. [9]

    Policy gradients with variance related risk criteria

    Dotan Di Castro, Aviv Tamar, and Shie Mannor. Policy gradients with variance related risk criteria. arXiv preprint arXiv:1206.6404, 2012

    2012 arXiv

  2. [10]

    Stevie Bergman, Shannon Spruit, Dirk Hovy, Y-Lan Boureau, and Verena Rieser

    Emily Dinan, Gavin Abercrombie, A. Stevie Bergman, Shannon Spruit, Dirk Hovy, Y-Lan Boureau, and Verena Rieser. Anticipating safety issues in e2e conversational AI : Framework and tooling, 2021. URL https://arxiv.org/abs/2107.03451

    2021 arXiv

  3. [11]

    Hashimoto

    Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2024. URL https://arxiv.org/abs/2305.14387

    2024 arXiv

  4. [12]

    Fundamentals of optimization theory with applications to machine learning

    Jean Gallier and Jocelyn Quaintance. Fundamentals of optimization theory with applications to machine learning. University of Pennsylvania Philadelphia, PA, 19104, 2019

    2019

  5. [13]

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned

    Deep Ganguli et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. ArXiv, abs/2209.07858, 2022. URL https://api.semanticscholar.org/CorpusID:252355458

    2022 arXiv

  6. [14]

    Scaling laws for reward model overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, 2022. URL https://api.semanticscholar.org/CorpusID:252992904

    2022

  7. [15]

    Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. In Findings, 2020. URL https://api.semanticscholar.org/CorpusID:221878771

    2020

  8. [16]

    Fairness guarantees under demographic shift

    Stephen Giguere, Blossom Metevier, Yuriy Brun, Bruno Castro Da Silva, Philip S Thomas, and Scott Niekum. Fairness guarantees under demographic shift. In Proceedings of the 10th International Conference on Learning Representations (ICLR), 2022

    2022

  9. [17]

    Improving alignment of dialogue agents via targeted human judgements, 2022

    Amelia Glaese et al. Improving alignment of dialogue agents via targeted human judgements, 2022. URL https://arxiv.org/abs/2209.14375

    2022 arXiv

  10. [18]

    The Llama 3 herd of models, 2024

    Aaron Grattafiori et al. The Llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

    2024 arXiv

  11. [19]

    Ethical challenges in data-driven dialogue systems, 2017

    Peter Henderson, Koustuv Sinha, Nicolas Angelard-Gontier, Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. Ethical challenges in data-driven dialogue systems, 2017. URL https://arxiv.org/abs/1711.09050

    2017 arXiv

  12. [20]

    Probability inequalities for sums of bounded random variables

    Wassily Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58 0 (301): 0 13--30, 1963

    1963

  13. [21]

    One-shot safety alignment for large language models via optimal dualization

    Xinmeng Huang, Shuo Li, Edgar Dobriban, Osbert Bastani, Hamed Hassani, and Dongsheng Ding. One-shot safety alignment for large language models via optimal dualization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview...

    2024

  14. [22]

    Jones, Shixiang Shane Gu, and Rosalind W

    Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, \`A gata Lapedriza, Noah J. Jones, Shixiang Shane Gu, and Rosalind W. Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. ArXiv, abs/1907.00456, 2019. URL https:...

    1907 arXiv

  15. [23]

    Beavertails: Towards improved safety alignment of LLM via a human-preference dataset, 2023

    Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Chi Zhang, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of LLM via a human-preference dataset, 2023. URL https://arxiv.org/abs/2307.04657

    2023

  16. [24]

    ChatGPT for good? O n opportunities and challenges of large language models for education

    Enkelejda Kasneci et al. ChatGPT for good? O n opportunities and challenges of large language models for education. Learning and Individual Differences, 2023. URL https://api.semanticscholar.org/CorpusID:257445349

    2023

  17. [25]

    GPT -4 passes the bar exam

    Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. GPT -4 passes the bar exam. Philosophical transactions. Series A, Mathematical, physical, and engineering sciences, 382, 2024. URL https://api.semanticscholar.org/CorpusID:257572753

    2024

  18. [26]

    Buy 4 reinforce samples, get a baseline for free! In DeepRLStructPred@ICLR, 2019

    Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 reinforce samples, get a baseline for free! In DeepRLStructPred@ICLR, 2019. URL https://api.semanticscholar.org/CorpusID:198489118

    2019

  19. [27]

    Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepa \ n o, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, and Victor Tseng

    Tiffany H. Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepa \ n o, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, and Victor Tseng. Performance of chatgpt on usmle: Potential for AI -assisted medical education using lar...

    2022

  20. [28]

    Enhancing llm safety via constrained direct preference optimization, 2024

    Zixuan Liu, Xiaolin Sun, and Zizhan Zheng. Enhancing llm safety via constrained direct preference optimization, 2024. URL https://arxiv.org/abs/2403.02475

    2024 arXiv

  21. [29]

    Offline contextual bandits with high probability fairness guarantees

    Blossom Metevier, Stephen Giguere, Sarah Brockman, Ari Kobren, Yuriy Brun, Emma Brunskill, and Philip S Thomas. Offline contextual bandits with high probability fairness guarantees. Advances in neural information processing systems, 32, 2019

    2019

  22. [30]

    Krumholz, Jure Leskovec, Eric J

    Michael Moor, Oishi Banerjee, Zahra F H Abad, Harlan M. Krumholz, Jure Leskovec, Eric J. Topol, and Pranav Rajpurkar. Foundation models for generalist medical artificial intelligence. Nature, 616: 0 259--265, 2023. URL https://api.semanticscholar.org/CorpusID:258083369

    2023

  23. [31]

    Rule based rewards for language model safety

    Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian D Kivlichan, Molly Lin, Alex Beutel, John Schulman, and Lilian Weng. Rule based rewards for language model safety. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024....

    2024

  24. [32]

    Training language models to follow instructions with human feedback

    Long Ouyang et al. Training language models to follow instructions with human feedback. ArXiv, abs/2203.02155, 2022. URL https://api.semanticscholar.org/CorpusID:246426909

    2022 arXiv

  25. [33]

    Enhancing safety in reinforcement learning with human feedback via rectified policy optimization, 2025

    Xiyue Peng, Hengquan Guo, Jiawei Zhang, Dongqing Zou, Ziyu Shao, Honghao Wei, and Xin Liu. Enhancing safety in reinforcement learning with human feedback via rectified policy optimization, 2025. URL https://arxiv.org/abs/2410.19933

    2025 arXiv

  26. [34]

    Qwen2.5 technical report, 2025

    Qwen et al. Qwen2.5 technical report, 2025. URL https://arxiv.org/abs/2412.15115

    2025 arXiv

  27. [35]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. ArXiv, abs/2305.18290, 2023. URL https://api.semanticscholar.org/CorpusID:258959321

    2023 arXiv

  28. [36]

    Sikchi, Joey Hejna, Bradley Knox, Chelsea Finn, and Scott Niekum

    Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit S. Sikchi, Joey Hejna, Bradley Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization in direct alignment algorithms. ArXiv, abs/2406.02900, 2024. URL https://api.semanticscholar.org/CorpusID:...

    2024 arXiv

  29. [37]

    Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards

    Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. In Thirty-seventh Conference on Neural Informat...

    2023

  30. [38]

    Simultaneous statistical inference

    G Rupert Jr et al. Simultaneous statistical inference. 2012

    2012

  31. [39]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv, abs/1707.06347, 2017. URL https://api.semanticscholar.org/CorpusID:28695052

    2017 arXiv

  32. [40]

    Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano

    Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022. URL https://arxiv.org/abs/2009.01325

    2022 arXiv

  33. [41]

    The probable error of a mean

    Student. The probable error of a mean. Biometrika, 6 0 (1): 0 1--25, 1908

    1908

  34. [42]

    Equilibrate RLHF : Towards balancing helpfulness-safety trade-off in large language models, 2025

    Yingshui Tan, Yilei Jiang, Yanshi Li, Jiaheng Liu, Xingyuan Bu, Wenbo Su, Xiangyu Yue, Xiaoyong Zhu, and Bo Zheng. Equilibrate RLHF : Towards balancing helpfulness-safety trade-off in large language models, 2025. URL https://arxiv.org/abs/2502.11555

    2025 arXiv

  35. [43]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford A lpaca: An instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca, 2023

    2023

  36. [44]

    Preventing undesirable behavior of intelligent machines

    Philip S Thomas, Bruno Castro da Silva, Andrew G Barto, Stephen Giguere, Yuriy Brun, and Emma Brunskill. Preventing undesirable behavior of intelligent machines. Science, 366 0 (6468): 0 999--1004, 2019

    2019

  37. [45]

    Lamda: Language models for dialog applications, 2022

    Romal Thoppilan et al. Lamda: Language models for dialog applications, 2022. URL https://arxiv.org/abs/2201.08239

    2022 arXiv

  38. [46]

    LlaMa 2: Open foundation and fine-tuned chat models, 2023

    Hugo Touvron et al. LlaMa 2: Open foundation and fine-tuned chat models, 2023. URL https://arxiv.org/abs/2307.09288

    2023 arXiv

  39. [47]

    Tran, Rei Sato, Takumi Tanabe, and Youhei Akimoto

    Akifumi Wachi, Thien Q. Tran, Rei Sato, Takumi Tanabe, and Youhei Akimoto. Stepwise alignment for constrained language model policy optimization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=VrVx83BkQX

    2024

  40. [48]

    Enforcing delayed-impact fairness guarantees

    Aline Weber, Blossom Metevier, Yuriy Brun, Philip S Thomas, and Bruno Castro da Silva. Enforcing delayed-impact fairness guarantees. arXiv preprint arXiv:2208.11744, 2022

    2022 arXiv

  41. [49]

    Ethical and social risks of harm from language models

    Laura Weidinger et al. Ethical and social risks of harm from language models. ArXiv, abs/2112.04359, 2021. URL https://api.semanticscholar.org/CorpusID:244954639

    2021 arXiv

  42. [50]

    Simple statistical gradient-following algorithms for connectionist reinforcement learning

    Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8: 0 229--256, 1992

    1992

  43. [51]

    Recipes for safety in open-domain chatbots, 2021

    Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. Recipes for safety in open-domain chatbots, 2021. URL https://arxiv.org/abs/2010.07079

    2021 arXiv

  44. [52]

    Qwen2 technical report, 2024

    An Yang et al. Qwen2 technical report, 2024. URL https://arxiv.org/abs/2407.10671

    2024 arXiv

  45. [53]

    A large language model for electronic health records

    Xi Yang et al. A large language model for electronic health records. NPJ Digital Medicine, 5, 2022. URL https://api.semanticscholar.org/CorpusID:255175535

    2022

  46. [54]

    Wenxuan Zhang, Philip H. S. Torr, Mohamed Elhoseiny, and Adel Bibi. Bi-factorial preference optimization: Balancing safety-helpfulness in language models, 2024. URL https://arxiv.org/abs/2408.15313

    2024 arXiv

  47. [55]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM -as-a- J udge with MT - B ench and C hatbot A rena, 2023 a . URL https://arxiv.org/ab...

    2023 arXiv

  48. [56]

    Secrets of RLHF in large language models part I : PPO

    Rui Zheng et al. Secrets of RLHF in large language models part I : PPO . ArXiv, abs/2307.04964, 2023 b . URL https://api.semanticscholar.org/CorpusID:259766568

    2023 arXiv

  49. [57]

    Ziegler, Seraphina Nix, Lawrence Chan, Tim Bauman, Peter Schmidt-Nielsen, Tao Lin, Adam Scherlis, Noa Nabeshima, Ben Weinstein-Raun, Daniel de Haas, Buck Shlegeris, and Nate Thomas

    Daniel M. Ziegler, Seraphina Nix, Lawrence Chan, Tim Bauman, Peter Schmidt-Nielsen, Tao Lin, Adam Scherlis, Noa Nabeshima, Ben Weinstein-Raun, Daniel de Haas, Buck Shlegeris, and Nate Thomas. Adversarial training for high-stakes reliability, 2022. URL https://arxiv.org/abs/2205.01663

    2022 arXiv

  50. [58]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.