Pith. sign in

REVIEW 1 cited by

A fixed ReLU penalty and a semantically labeled cost model are proposed to make RLHF safer, but the 'certifiable' guarantee is not fully supported.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 11:40 UTC pith:6T453XTW

load-bearing objection The empirical story is credible, but the 'certifiable' guarantee is vacuous as written: the safety threshold d is negative while costs are sigmoid probabilities in [0,1], so the feasible set is empty.

arxiv 2510.03520 v2 pith:6T453XTW submitted 2025-10-03 cs.LG cs.AIcs.SYeess.SY

Certifiable Safe RLHF: Semantic Grounding and Fixed Penalty Constraint Optimization for Safer LLM Alignment

Kartik Pandit , Sourav Ganguly , Arnesh Banerjee , Shaahin Angizi , Arnob Ghosh This is my paper
classification cs.LG cs.AIcs.SYeess.SY
keywords penaltysafetycs-rlhfmodeloptimizationcertifiableconstrainedconstraint
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

LLMs are trained to be helpful with reinforcement learning from human feedback (RLHF). Safety is usually handled as a constraint: keep a harmfulness score below a threshold, and adjust a Lagrange multiplier during training to balance helpfulness and safety. CS-RLHF replaces that multiplier with a fixed penalty: only when the average harmfulness score exceeds the threshold does a penalty term switch on, pushing the model back toward safe responses. The authors also train a separate cost model on roughly 1,500 prompt-response pairs labeled safe or unsafe, instead of preference pairs, so that harm is judged by meaning rather than by keywords.

The paper claims two benefits: less hyperparameter tuning and a 'certifiable' guarantee that an optimal policy is nearly safe and at least as helpful as the constrained optimum. The empirical part compares CS-RLHF with Safe-RLHF on regular prompts, jailbreak prompts, best-of-N sampling, and a small set of multi-turn attacks, reporting more safe responses and better alignment with human labels.

The main concerns are in the theory-to-practice link. The theorem assumes exact optimality of the population objective, but training uses a PPO surrogate over minibatches. The proof also needs unstated conditions, and the safety threshold in the hyperparameter table appears inconsistent with the sigmoid cost scores. The evaluations are small and use in-house human labels. The fixed-penalty idea is plausible and standard in constrained optimization, but the 'certifiable' label is not supported as written.

Core claim

The load-bearing assertion is in Section 4.1: 'an optimal solution of (5) has a better reward while ensuring that the constraint violation is bounded by ε... such a guarantee cannot be achieved for a Lagrangian-based approach.' Formally, Theorem 1 states that for λ≥Rmax/ε, any optimal πhat of max J_R−λReLU(J_C) satisfies J_R(πhat)≥J_R(π*) and J_C(πhat)≤d+ε, and Corollary 1 extends this to best-of-N decode-time selection. If true, CS-RLHF would give fixed-penalty safe RLHF a provable feasibility guarantee without dual tuning.

Load-bearing premise

The theory-to-training bridge: Theorem 1 and Corollary 1 certify only an exact maximizer of the population objective (5), but the actual algorithm minimizes the PPO surrogate L_CS (Eq. 38) with a minibatch indicator I{Ĵ_C>0}, and no result shows the trained policy is an ε-optimal solution of (5). Relatedly, Theorem 1's proof silently assumes J_R(π*_θ)≥0 and Corollary 1's proof assumes u(x,y_j)≥0 for the safe candidate; without these, the contradiction arguments fail. If PPO stalls or the learned cost model mislabels harmful content, the claimed certificate does not apply.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

The central theoretical result rests on boundedness and exact optimality assumptions; the empirical safety claim rests on the learned cost model faithfully measuring harm and on in-house human labels. The unstated nonnegativity assumptions in the proofs are the most fragile.

free parameters (4)
  • Fixed penalty weight λ = 20.0 (Table 5)
    Central to Theorem 1; requires λ≥Rmax/ε but Rmax/ε not specified. Appendix B shows λ=3–10 gives weak enforcement and λ=30 suppresses helpfulness, so λ is effectively tuned.
  • Safety threshold d = -0.4, -0.5 (Table 5)
    Constraint threshold in (2)/(4). As written it is negative while cψ∈[0,1] (Eq. 9), making the constraint infeasible; likely intended d≈0.5 given 'safe region c<0.5'. This inconsistency affects Theorem 1's applicability.
  • KL regularization weight β = 0.1 (Table 5)
    Standard RLHF hyperparameter in Eq. (1)/(33); chosen by hand and affects the helpfulness/safety trade-off.
  • PTX coefficient γ = 16.0 (Table 5)
    Supervised regularization coefficient in Eq. (30)/(38); chosen by hand to prevent catastrophic forgetting and part of the final training objective.
axioms (6)
  • domain assumption Reward and cost are bounded: |r|≤Rmax, |c|≤Cmax.
    Used in Theorem 1, Corollary 1, and Lemma 1. Bounds are asserted but Rmax/Cmax values are never given or verified for the learned models.
  • ad hoc to paper Constrained optimum has nonnegative reward J_R(π*)≥0 and safe BoN candidate has u≥0.
    Unstated in Appendix D and Section 6; needed for the contradiction steps in Theorem 1 and Corollary 1.
  • ad hoc to paper The batch-level PPO objective (38) is a faithful surrogate for (5), and PPO returns an (approximate) global optimum.
    No convergence or optimality theorem connects the implemented loss to the certified population objective.
  • domain assumption The learned cost model faithfully measures true harmfulness / semantic intent.
    The safety certificate is w.r.t. cψ; if cψ is miscalibrated, unsafe responses can pass. Appendix A concedes limited coverage.
  • domain assumption πref and candidate distributions satisfy the coverage/smoothness conditions for the soft-BoN regret bounds.
    Lemmas 2–3 and Theorem 2 inherit Assumptions 1–2 and coverage definitions from Aminian et al.; no verification is provided on the actual model.
  • domain assumption A global optimum πhat of the nonconvex objective (5) exists.
    Neural policies and nonconvex objectives do not guarantee global optima; the theorem assumes one is found.

pith-pipeline@v1.3.0-alltime-deepseek · 29776 in / 16563 out tokens · 202075 ms · 2026-08-04T11:40:04.430392+00:00 · methodology

0 comments
read the original abstract

Ensuring safety is a foundational requirement for large language models (LLMs). Achieving an appropriate balance between enhancing the utility of model outputs and mitigating their potential for harm is a complex and persistent challenge. Contemporary approaches frequently formalize this problem within the framework of Constrained Markov Decision Processes (CMDPs) and employ established CMDP optimization techniques. However, these methods exhibit two notable limitations. First, their reliance on reward and cost functions renders performance highly sensitive to the underlying scoring mechanism, which must capture semantic meaning rather than being triggered by superficial keywords. Second, CMDP-based training entails tuning dual-variable, a process that is both computationally expensive and does not provide any provable safety guarantee for a fixed dual variable that can be exploitable through adversarial jailbreaks. To overcome these limitations, we introduce Certifiable Safe-RLHF (CS-RLHF) that introduces a cost model trained on a large-scale corpus to assign semantically grounded safety scores. In contrast to the lagrangian-based approach, CS-RLHF adopts a rectified penalty-based formulation. This design draws on the theory of exact penalty functions in constrained optimization, wherein constraint satisfaction is enforced directly through a suitably chosen penalty term. With an appropriately scaled penalty, feasibility of the safety constraints can be guaranteed at the optimizer, eliminating the need for dual-variable updates. Empirical evaluation demonstrates that CS-RLHF outperforms state-of-the-art LLM model responses rendering at-least 5 times efficient against nominal and jail-breaking prompts

Figures

Figures reproduced from arXiv: 2510.03520 by Arnesh Banerjee, Arnob Ghosh, Kartik Pandit, Shaahin Angizi, Sourav Ganguly.

Figure 1
Figure 1. Figure 1: Comparison between RLHF, Safe-RLHF and CS-RLHF frameworks 2 PRELIMINARIES AND BACKGROUND In this section, we discuss the necessary details that builds the foundation for our model (CS-RLHF). We begin with RLHF, a framework for aligning large language models with human preferences followed by incorporating safety in RLHF using Safe-RLHF. We finally conclude this section with a brief discussion of jailbreak … view at source ↗
Figure 2
Figure 2. Figure 2: Cost–reward trade-off comparison (a) Alpaca-7B(left), (b) Safe-RLHF(middle), and (c) CS￾RLHF(right). Black dots denote jailbreak responses. Green and white shades denotes Safe and Unsafe zone respectively. Ideally, the scatter plot of cost versus reward should be highly concentrated in the second quadrant (high reward, low cost). As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Safety against 40 unseen jailbreak prompts. CS-RLHF restricts unsafe generations to 15%, signifi￾cantly lower than Safe-RLHF and other models. 6 INFERENCE TIME SAFETY A common inference strategy is Best-of-N (BoN) sampling. We adopt an analogous procedure that mirrors our surrogate objective: max π EY ∼π(·|x) [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Reward–cost distributions under alternative [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Scores assigned by Safe-RLHF and CS-RLHF cost models on identical prompt–response pairs. Both models were given the same inputs; the figures highlights differences in scoring behavior. Excerpts are shown for brevity, with full responses and additional examples in Appendix J( [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Optimistic Policy Learning under Pessimistic Adversaries with Regret and Violation Guarantees

    cs.LG 2026-04 unverdicted novelty 8.0

    RHC-UCRL is the first algorithm for safety-constrained RL under explicit adversarial dynamics, providing sub-linear regret and constraint violation guarantees by maintaining optimism over both agent and adversary policies.

Reference graph

Works this paper leans on

56 extracted references · 28 linked inside Pith · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Constrained Markov decision processes

    Eitan Altman. Constrained Markov decision processes. Routledge, 2021

    2021

  3. [3]

    Best-of-n through the smoothing lens: Kl divergence and regret analysis

    Gholamali Aminian, Idan Shenfeld, Amir R Asadi, Ahmad Beirami, and Youssef Mroueh. Best-of-n through the smoothing lens: Kl divergence and regret analysis. arXiv preprint arXiv:2507.05913, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Concrete problems in ai safety

    Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Man \'e . Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016

    Pith/arXiv arXiv 2016

  5. [5]

    A general language assistant as a laboratory for alignment

    Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021

    Pith/arXiv arXiv 2021

  6. [6]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022

    Pith/arXiv arXiv 2022

  7. [7]

    Nonlinear programming

    Dimitri P Bertsekas. Nonlinear programming. Journal of the Operational Research Society, 48 0 (3): 0 334--334, 1997

    1997

  8. [8]

    Measuring progress on scalable oversight for large language models

    Samuel R Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamil \.e Luko s i \=u t \.e , Amanda Askell, Andy Jones, Anna Chen, et al. Measuring progress on scalable oversight for large language models. arXiv preprint arXiv:2211.03540, 2022

    Pith/arXiv arXiv 2022

  9. [9]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952

    1952

  10. [10]

    Large language model in creative work: The role of collaboration modality and user expertise

    Zenan Chen and Jason Chan. Large language model in creative work: The role of collaboration modality and user expertise. Management Science, 70 0 (12): 0 9101--9117, 2024

    2024

  11. [11]

    Risk-sensitive and data-driven sequential decision making

    Yinlam Chow. Risk-sensitive and data-driven sequential decision making. PhD thesis, Stanford University, 2017

    2017

  12. [12]

    Deep reinforcement learning from human preferences

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017

    2017

  13. [13]

    Safe RLHF : Safe reinforcement learning from human feedback

    Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF : Safe reinforcement learning from human feedback. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=TyFrPOKYXw

    2024

  14. [14]

    Toxicity in chatgpt: Analyzing persona-assigned language models

    Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. Toxicity in chatgpt: Analyzing persona-assigned language models. arXiv preprint arXiv:2304.05335, 2023

    Pith/arXiv arXiv 2023

  15. [15]

    On the creativity of large language models

    Giorgio Franceschelli and Mirco Musolesi. On the creativity of large language models. AI & SOCIETY, pp.\ 1--11, 2024

    2024

  16. [16]

    Artificial intelligence, values, and alignment

    Iason Gabriel. Artificial intelligence, values, and alignment. Minds and Machines, 30 0 (3): 0 411--437, 2020

    2020

  17. [17]

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned

    Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858, 2022

    Pith/arXiv arXiv 2022

  18. [18]

    Realtoxicityprompts: Evaluating neural toxic degeneration in language models

    Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462, 2020

    Pith/arXiv arXiv 2009

  19. [19]

    Multimodal-gpt: A vision and language model for dialogue with humans

    Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-gpt: A vision and language model for dialogue with humans. arXiv preprint arXiv:2305.04790, 2023

    Pith/arXiv arXiv 2023

  20. [20]

    Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models, 2023

    Abhijit Guha et al. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models, 2023

    2023

  21. [21]

    Is best-of-n the best of them? coverage, scaling, and optimality in inference-time alignment

    Audrey Huang, Adam Block, Qinghua Liu, Nan Jiang, Akshay Krishnamurthy, and Dylan J Foster. Is best-of-n the best of them? coverage, scaling, and optimality in inference-time alignment. arXiv preprint arXiv:2503.21878, 2025

    Pith/arXiv arXiv 2025

  22. [22]

    Beavertails: Towards improved safety alignment of llm via a human-preference dataset

    Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. Advances in Neural Information Processing Systems, 36: 0 24678--24704, 2023

    2023

  23. [23]

    Advwave: Stealthy adversarial jailbreak attack against large audio-language models

    Mintong Kang, Chejian Xu, and Bo Li. Advwave: Stealthy adversarial jailbreak attack against large audio-language models. arXiv preprint arXiv:2412.08608, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    Performance of chatgpt on usmle: potential for ai-assisted medical education using large language models

    Tiffany H Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepa \ n o, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et al. Performance of chatgpt on usmle: potential for ai-assisted medical education using large language models. PLoS digital health, 2 0 (2): 0 e0000198, 2023

    2023

  25. [25]

    Open sesame! universal black-box jailbreaking of large language models

    Rotem Lapid, Yftah Ziser, Jonathan Katz, Yoav Goldberg, and Yonatan Belinkov. Open sesame! universal black-box jailbreaking of large language models. arXiv preprint arXiv:2309.01446, 2023. URL https://arxiv .org/abs/2309.01446

    Pith/arXiv arXiv 2023

  26. [26]

    Llm defenses are not robust to multi-turn human jailbreaks yet, 2024

    Nathaniel Li, Ziwen Han, Ian Steneker, Willow Primack, Riley Goodside, Hugh Zhang, Zifan Wang, Cristina Menghini, and Summer Yue. Llm defenses are not robust to multi-turn human jailbreaks yet, 2024. URL https://arxiv.org/abs/2408.15221

    Pith/arXiv arXiv 2024

  27. [27]

    Detam: Defending llms against jailbreak attacks via targeted attention modification

    Yu Li, Han Jiang, and Zhihua Wei. Detam: Defending llms against jailbreak attacks via targeted attention modification. In Findings of the Association for Computational Linguistics: ACL 2025, 2025. URL https://aclanthology.org/2025.findings-acl.613

    2025

  28. [28]

    Llm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models

    Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. Llm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. arXiv preprint arXiv:2305.13655, 2023

    Pith/arXiv arXiv 2023

  29. [29]

    A survey of attacks on large vision--language models: Resources, advances, and future trends

    Daizong Liu, Mingyu Yang, Xiaoye Qu, Pan Zhou, Yu Cheng, and Wei Hu. A survey of attacks on large vision--language models: Resources, advances, and future trends. IEEE Transactions on Neural Networks and Learning Systems, 2025

    2025

  30. [30]

    Formalizing and benchmarking prompt injection attacks and defenses

    Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. Formalizing and benchmarking prompt injection attacks and defenses. In 33rd USENIX Security Symposium (USENIX Security 24), pp.\ 1831--1847, 2024

    2024

  31. [31]

    Jailbreakv: A benchmark for assessing the robustness of multimodal large language models against jailbreak attacks

    Weidi Luo, Siyuan Ma, Xiaogeng Liu, Xiaoyu Guo, and Chaowei Xiao. Jailbreakv: A benchmark for assessing the robustness of multimodal large language models against jailbreak attacks. arXiv preprint arXiv:2404.03027, 2024

    Pith/arXiv arXiv 2024

  32. [32]

    Foundation models for generalist medical artificial intelligence

    Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, and Pranav Rajpurkar. Foundation models for generalist medical artificial intelligence. Nature, 616 0 (7956): 0 259--265, 2023

    2023

  33. [33]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022

    2022

  34. [34]

    Autotutor meets large language models: A language model tutor with rich pedagogy and guardrails

    Sankalan Pal Chowdhury, Vil \'e m Zouhar, and Mrinmaya Sachan. Autotutor meets large language models: A language model tutor with rich pedagogy and guardrails. In Proceedings of the Eleventh ACM Conference on Learning@ Scale, pp.\ 5--15, 2024

    2024

  35. [35]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  36. [36]

    Characteristics of harmful text: Towards rigorous benchmarking of language models

    Maribeth Rauh, John Mellor, Jonathan Uesato, Po-Sen Huang, Johannes Welbl, Laura Weidinger, Sumanth Dathathri, Amelia Glaese, Geoffrey Irving, Iason Gabriel, et al. Characteristics of harmful text: Towards rigorous benchmarking of language models. Advances in Neural Information Processing Systems, 35: 0 24720--24739, 2022

    2022

  37. [37]

    Smoothllm: Defending large language models against jailbreaking attacks

    Alexander Robey, Eric Wong, Hamed Hassani, and George J Pappas. Smoothllm: Defending large language models against jailbreaking attacks. arXiv preprint arXiv:2310.03684, 2023

    Pith/arXiv arXiv 2023

  38. [38]

    High-dimensional continuous control using generalized advantage estimation

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015

    Pith/arXiv arXiv 2015

  39. [39]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  40. [40]

    Large language model safety: A holistic survey

    Dan Shi, Tianhao Shen, Yufei Huang, Zhigen Li, Yongqi Leng, Renren Jin, Chuang Liu, Xinwei Wu, Zishan Guo, Linhao Yu, et al. Large language model safety: A holistic survey. arXiv preprint arXiv:2412.17686, 2024

    Pith/arXiv arXiv 2024

  41. [41]

    On pi controllers for updating lagrange multipliers in constrained optimization

    Motahareh Sohrabi, Juan Ramirez, Tianyue H Zhang, Simon Lacoste-Julien, and Jose Gallego-Posada. On pi controllers for updating lagrange multipliers in constrained optimization. arXiv preprint arXiv:2406.04558, 2024

    Pith/arXiv arXiv 2024

  42. [42]

    Learning to summarize with human feedback

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33: 0 3008--3021, 2020

    2020

  43. [43]

    Responsive safety in reinforcement learning by pid lagrangian methods

    Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pp.\ 9133--9143. PMLR, 2020

    2020

  44. [44]

    Safety assessment of chinese large language models

    Hao Sun, Zhexin Zhang, Jiawen Deng, Jiale Cheng, and Minlie Huang. Safety assessment of chinese large language models. arXiv preprint arXiv:2304.10436, 2023

    Pith/arXiv arXiv 2023

  45. [45]

    Data feedback loops: Model-driven amplification of dataset biases

    Rohan Taori and Tatsunori Hashimoto. Data feedback loops: Model-driven amplification of dataset biases. In International Conference on Machine Learning, pp.\ 33883--33920. PMLR, 2023

    2023

  46. [46]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Amjad Albert, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Amar Bhargava, Shruti Bhosale, Alban Desmaison Bressand, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. URL https://arxiv.org/abs/2307.09288

    Pith/arXiv arXiv 2023

  47. [47]

    Large language models in cryptocurrency securities cases: can a gpt model meaningfully assist lawyers? Artificial Intelligence and Law, pp.\ 1--47, 2024

    Arianna Trozze, Toby Davies, and Bennett Kleinberg. Large language models in cryptocurrency securities cases: can a gpt model meaningfully assist lawyers? Artificial Intelligence and Law, pp.\ 1--47, 2024

    2024

  48. [48]

    Yu, and Qingsong Wen

    Shen Wang, Tianlong Xu, Hang Li, Chaoli Zhang, Joleen Liang, Jiliang Tang, Philip S. Yu, and Qingsong Wen. Large language models for education: A survey and outlook, 2024

    2024

  49. [49]

    Jailbroken: How does llm safety training fail? In NeurIPS, 2023

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail? In NeurIPS, 2023. URL https://arxiv.org/abs/2307.02483

    Pith/arXiv arXiv 2023

  50. [50]

    Ethical and social risks of harm from language models

    Laura Weidinger, John Mellor, Maribeth Rauh, et al. Ethical and social risks of harm from language models. arXiv, 2021

    2021

  51. [51]

    Taxonomy of risks posed by language models

    Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, et al. Taxonomy of risks posed by language models. In Proceedings of the 2022 ACM conference on fairness, accountability, and transparency, pp.\ 214--229, 2022

    2022

  52. [52]

    Enhancing semantic consistency of large language models through model editing: An interpretability-oriented approach

    Jingyuan Yang, Dapeng Chen, Yajing Sun, Rongjun Li, Zhiyong Feng, and Wei Peng. Enhancing semantic consistency of large language models through model editing: An interpretability-oriented approach. arXiv preprint arXiv:2501.11041, 2025

    Pith/arXiv arXiv 2025

  53. [53]

    Conservative distributional reinforcement learning with safety constraints

    Hengrui Zhang, Youfang Lin, Sheng Han, Shuo Wang, and Kai Lv. Conservative distributional reinforcement learning with safety constraints. arXiv preprint arXiv:2201.07286, 2022

    Pith/arXiv arXiv 2022

  54. [54]

    Improving the robustness of large language models via consistency alignment

    Yukun Zhao, Lingyong Yan, Weiwei Sun, Guoliang Xing, Shuaiqiang Wang, Chong Meng, Zhicong Cheng, Zhaochun Ren, and Dawei Yin. Improving the robustness of large language models via consistency alignment. arXiv preprint arXiv:2403.14221, 2024

    Pith/arXiv arXiv 2024

  55. [55]

    Archer: Training language model agents via hierarchical multi-turn rl

    Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-turn rl. arXiv preprint arXiv:2402.19446, 2024

    Pith/arXiv arXiv 2024

  56. [56]

    Universal and transferable adversarial attacks on aligned language models

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023

    Pith/arXiv arXiv 2023