Pith. sign in

REVIEW 3 major objections 5 minor 60 references

Refining Positive and Toxic Samples for Dual Safety Self-Alignment of LLMs with Minimal Human Interventions

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fewer than 50 human annotations, plus self-generated toxic pairs, can safety-align an LLM.

desk verdict Solid low-annotation safety alignment with a real helpfulness-measurement gap; worth refereeing if the authors add an open-ended instruction-following eval. read the letter →

arxiv 2502.08657 v1 pith:RYCS6RI2 submitted 2025-02-08 cs.CL cs.AI

Jingxin Xu , Guoshun Nan , Sheng Guan , Sicong Leng , Yilian Liu , Zixiao Wang , Yuyang Ma , Zhili Zhou
show 2 more authors
Yanzhao Hou Xiaofeng Tao
This is my paper · ORCID
classification cs.CLcs.AI
keywords safetyalignmentself-alignmenttoxicsamplesunlikelihoodtraininglargelanguagemodelsminimalhumansupervisionjailbreakrobustnessinstructiontuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PT-ALIGN claims that an open-weight LLM can be safety-aligned almost entirely by itself: with fewer than 50 human-written annotations, the model generates its own safety instructions and, for each one, a paired harmless response and a deliberately toxic response. Training on these pairs with a maximum-likelihood loss on the harmless text plus a token-level unlikelihood loss on the toxic text is said to move the model away from harmful output while keeping helpfulness intact. On LLaMA2-13B the paper reports the PKU-SafeRLHF harm score dropping from 2.21 to 0.30, the HHH harmless score rising from 74.14% to 98.28%, an AutoDAN attack success rate of 0.51% (versus 5.06% for LLaMA2-13B-chat), and roughly flat scores on MMLU, TruthfulQA, and SocialQA. The practical stake is that safety tuning no longer needs large curated preference datasets or RLHF-scale human feedback.

What carries the argument

The load-bearing object is the instruction-positive-negative triplet $(i,p,n)$, with the loss $$L = -\frac{1}{N}\left(\sum_{\$\beta$}\log \mathrm{Pr}_\$\theta$(p_\$\beta$ \mid p_{<\$\beta$}, i) + \$\lambda$ \sum_{\gamma} I(\gamma \ge \gamma^*)\log(1 - \mathrm{Pr}_\$\theta$(n_\gamma \mid n_{<\gamma}, i))\right),$$ where $\mathrm{Pr}_\theta$ is the model's next-token probability, $\gamma^*$ is the first token position where the positive and toxic responses disagree, and $I(\cdot)$ is the indicator function. The MLE term teaches the model to continue harmless text; the unlikelihood term suppresses the toxic continuation at token level; the fine-grained mask keeps tokens shared with the positive response from being penalized. Around this core, the pipeline uses self-constraint continuation (the model extends a short human-written rule into a full constraint prompt) and in-context seeds to generate the polarized pairs without per-sample human labels.

What would settle it

Have fresh human annotators score a random sample of the 16,020 generated instruction-response triplets for correctness of the positivity and toxicity labels; if a substantial share of the 'toxic' responses are actually benign refusals, or the 'harmless' responses contain subtle harmful content, the dual-polarity signal the loss is designed to exploit is not present.

Watch

Extended reading notes

Core claim

The paper's central claim is that highly polarized instruction-response triplets, generated by the model under its own constraints, give a better safety signal than conventional preference pairs. Starting from ten safety domains, the LLM subdivides them into topics, synthesizes a large instruction set, extends a short human-written rule into a full self-constraint prompt, and then uses those constraints plus a few seed examples to write both a harmless answer $p$ and a severely toxic answer $n$ for each instruction $i$. The resulting triplets $(i,p,n)$ are used for one epoch of LoRA fine-tuning with a combined loss: MLE on $p$ and fine-grained unlikelihood on $n$, where the unlikelihood term is masked to apply only from the first token at which $p$ and $n$ diverge. Because the two responses are far apart in safety polarity, the model can learn to refuse harmful requests directly, and the mask prevents the shared harmless prefix from being penalized. Across nine open-weight models the reported effect is a large uniform improvement on safety benchmarks with small changes on general-knowledge benchmarks, and a lower AutoDAN jailbreak success rate than an official chat-tuned baseline.

Load-bearing premise

The argument rests on the premise that an LLM can write its own safety constraints and then generate, for the same instruction, a genuinely harmless and a genuinely severely toxic response, so that the two losses push in exactly opposite directions without any per-sample human verification.

Editorial extensions

If this is right

  • Safety alignment becomes a near-automatic post-training step for open-weight LLMs: the same loop of self-generated instructions, polarized responses, and dual loss can be rerun for new safety categories without building a new human-labeled dataset.
  • Toxic outputs, normally discarded during data cleaning, are repositioned as a reusable supervisory signal, so future pipelines can deliberately collect extreme negative examples.
  • Fine-grained token-level unlikelihood with a shared-prefix mask offers an alternative to preference-based objectives such as DPO and KTO, and the paper reports smoother, faster-converging training loss curves for it.
  • The reported scaling behavior implies that generating more positive/toxic triplets raises harmlessness further, so teams can trade annotation effort for synthetic dataset size.
  • The low AutoDAN attack success rate compared with the official chat-tuned model suggests the trained refusals survive prompt paraphrasing and iterative jailbreak rewrites.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the recipe is driven by the model's own constraints rather than a fixed taxonomy, it should transfer to custom safety policies: an organization could swap the ten usage-policy domains for its own content rules and keep the same dual-loss loop, a step the paper does not test.
  • The design leaves open whether the fine-grained mask or the toxic samples themselves carries the safety gain; an ablation that trains with toxic pairs but no mask, and with a mask but random negative text, would isolate the mechanism.
  • The most plausible side effect of a strong unlikelihood penalty is over-refusal on benign instructions that lexically resemble the toxic set, so a natural extension is to measure refusal rates on harmless prompts with partial overlap with toxic content.
  • The triplet loss could serve as a warm start for preference optimization: initializing with PT-ALIGN and then applying DPO or KTO on human preferences might combine the sharp polarity signal with nuanced ranking, an order the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents PT-ALIGN, a safety self-alignment method that requires fewer than 50 human annotations. It uses an LLM to generate safety topics, instructions, and paired positive/toxic responses, then fine-tunes with MLE on positive responses and a fine-grained unlikelihood loss on toxic responses, skipping the shared token prefix to avoid over-penalizing common phrasing. Experiments on 9 open-source LLMs report large safety improvements on HHH, PKU-SafeRLHF, and AutoDAN, with small changes in MMLU, TruthfulQA, and SocialQA.

Significance. If the helpfulness-preservation claim holds, PT-ALIGN is a strong result: nearly annotation-free, effective across multiple model families, and with an internal ablation showing the fine-grained UT variant outperforms the non-fine-grained one. The paper also provides scaling and seed-robustness analyses and compares DPO/KTO variants on the same generated data. These strengths make the core contribution credible. However, the helpfulness half of the central claim rests on a narrow evaluation basis, so the significance is conditional on further evidence.

major comments (3)
  1. [Section IV.C, Table II] The claim that PT-ALIGN 'does not compromise helpfulness' and 'decouples safety from effectiveness' is supported only by the HHH helpfulness metric, which is a small multiple-choice set (about 300 items) where a refusal can be scored as helpful. The general-capability results in Table V are also multiple-choice probes (MMLU, TruthfulQA, SocialQA) and show consistent small declines without error bars. Please add an open-ended helpfulness evaluation (e.g., MT-Bench or AlpacaEval) or a human helpfulness rating on a held-out set, and report variance across seeds, before the decoupling claim can be accepted.
  2. [Section IV.A(d), Table II] The headline harm score decrease from 2.21 to 0.30 is produced by the 'absolute-harmfulness-predictor-redteam' model from prior work, not by fresh human annotation. Since this metric is load-bearing for the abstract's headline, the paper should either validate the harm scores on a human-annotated sample or relegate the model-based scores to secondary status and rely on the other safety metrics.
  3. [Section III.C, Equations (3)-(4)] The fine-grained UT loss applies the unlikelihood penalty to every token of the toxic response starting from the first token that differs from the positive response, not only to tokens that are actually harmful. Consequently, the method penalizes harmless divergent tokens (e.g., alternative refusal phrasings), and the claim that it 'minimizes the output of harmful words at the token level' is imprecise. The ablation in Table VIII shows the shared-prefix skip helps on HHH, but because HHH is a compact multiple-choice set, this does not establish that the penalty is decoupled from helpfulness in open-ended generation.
minor comments (5)
  1. [Various] There are typos: 'PN-ALIGN' appears in Table VI and in Section IV.A(b); 'Table 6' should be 'Table VI'; 'Gmini-1.0-pro' in Section IV.J.
  2. [Equations (1) and (3)] Equations (1) and (3) use 'X' where a summation sign is intended, and the normalization by N in Eq. (1) is ambiguous because the sum index β is also used for the sequence length.
  3. [Table II] Table II's column header 'pku-preference' is inconsistent with the text's 'PKU-SafeRLHF Preference'; please unify terminology.
  4. [Section IV.A] The list of the nine evaluated models is spread across Tables II and III; a consolidated model list in the setup would improve readability.
  5. [Figure 4] Figure 4's legend does not identify which line is the baseline dashed line; please add a label.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PT-ALIGN's safety and helpfulness claims rest on external benchmarks and a standard MLE-plus-unlikelihood objective, not on fitted or self-referential evaluation signals.

full rationale

The claimed derivation chain is not circular. PT-ALIGN synthesizes a safety dataset by having the target LLM generate topics, instructions, and paired positive/toxic responses under constraints (Section III-B), then trains with Equation (3), which combines MLE on positive samples with fine-grained token-level unlikelihood on toxic samples, using a fixed penalty coefficient λ = 0.4 (Sections III-C, IV-A). The reported safety gains are measured on external benchmarks not used in data synthesis or training: BIG-bench HHH, PKU-SafeRLHF Preference/Evaluation, Safety Evaluation Instruction Datasets, and AutoDAN within HarmBench (Section IV-A-d). The helpfulness and general-capability claims are likewise tested on HHH, TruthfulQA, SocialQA, and MMLU, none of which are used to filter or fit the training samples. The absolute-harmfulness-predictor-redteam scorer is taken from prior external work ([13], [32]) and is not a parameter fitted by this paper. There is no load-bearing self-citation: the references invoked for the evaluation protocol and unlikelihood method are external (Bai et al.; Bianchi et al.; Welleck et al.), and no uniqueness theorem or prior same-author result is used to force the approach. The strongest residual concern identified by the reader — that HHH multiple-choice helpfulness may reward refusal-heavy responses — is a benchmark-validity or measurement issue, not a circularity issue, because the benchmark is independent of the training objective and is not used as a training signal. The paper also states that training data were checked for non-overlap with test data, further reducing contamination risk. No equation reduces to its own input, no fitted quantity is renamed as a prediction, and no central claim is equivalent by construction to an assumption.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

PT-ALIGN introduces no new physical or architectural entities; its contributions are a data-synthesis protocol and a loss modification. The main burdens are the hand-chosen hyperparameters (lambda, seed counts), the domain assumption that self-generated toxic samples are valid supervision, and the reliance on saturation-prone safety benchmarks.

free parameters (3)
  • UT penalty coefficient lambda = 0.4
    Selected by scanning [0.1, 0.4, 0.7, 1.0] on one model (LLaMA2-13B, Table IX). The paper reports the best value on the development/eval setup, so it is a hyperparameter fitted to the evaluation setting.
  • Number of safety topics per domain (10) = 10 topics per domain
    Chosen by hand to shape the synthetic instruction distribution; no sensitivity analysis is provided.
  • Seed examples for instruction generation (10) and response annotation (6) = 10 instruction seeds; 6 response seeds
    Chosen by hand. The paper shows three different seed sets give roughly similar results on Alpaca-13B but does not vary the number of seeds.
assumptions (4)
  • domain assumption The base model can generate sufficiently reliable positive and toxic responses under self-written constraints and ICL seeds.
    The entire PT-ALIGN pipeline depends on the target LLM being able to synthesize high-quality triplets. The paper notes that the model occasionally generates non-harmful instructions that stabilize helpfulness, implicitly conceding imperfect control over synthesis quality.
  • ad hoc to paper The shared-prefix skip (Equation 4) is the correct correction for unlikelihood over-penalization.
    The paper motivates the prefix-skip mechanism with the observation that positive and toxic responses share words, but provides no formal or empirical analysis of when token-level penalties hurt rather than help; the ablation only shows the combined variant is better on one model.
  • domain assumption HHH multiple-choice accuracy and the automated harmfulness classifier are valid proxies for real-world safety.
    The paper's main safety claims are measured with the compact HHH set and PKU-SafeRLHF, plus a red-team harm score model. These are known to saturate and to be gaming-prone; the paper's own admission that helpfulness scores may be inflated by refusal-style answers reinforces this concern.
  • domain assumption Gradient penalties on harmful tokens do not degrade general capabilities more than the reported single-run numbers suggest.
    The preserved-helpfulness claim is based on one run per model on TruthfulQA, SocialQA, and MMLU, with no error bars or multiple seeds, so small average declines could be noise or could be real degradation that the evaluation lacks power to detect.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Refining Positive and Toxic Samples for Dual Safety Self-Alignment of LLMs with Minimal Human Interventions." pith.science (2026). https://pith.science/paper/RYCS6RI2

@misc{pith2026250208657,
  author       = {Pith},
  title        = {Pith review of: Refining Positive and Toxic Samples for Dual Safety Self-Alignment of LLMs with Minimal Human Interventions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RYCS6RI2}},
  note         = {Machine review of arXiv:2502.08657}
}
read the original abstract

Recent AI agents, such as ChatGPT and LLaMA, primarily rely on instruction tuning and reinforcement learning to calibrate the output of large language models (LLMs) with human intentions, ensuring the outputs are harmless and helpful. Existing methods heavily depend on the manual annotation of high-quality positive samples, while contending with issues such as noisy labels and minimal distinctions between preferred and dispreferred response data. However, readily available toxic samples with clear safety distinctions are often filtered out, removing valuable negative references that could aid LLMs in safety alignment. In response, we propose PT-ALIGN, a novel safety self-alignment approach that minimizes human supervision by automatically refining positive and toxic samples and performing fine-grained dual instruction tuning. Positive samples are harmless responses, while toxic samples deliberately contain extremely harmful content, serving as a new supervisory signals. Specifically, we utilize LLM itself to iteratively generate and refine training instances by only exploring fewer than 50 human annotations. We then employ two losses, i.e., maximum likelihood estimation (MLE) and fine-grained unlikelihood training (UT), to jointly learn to enhance the LLM's safety. The MLE loss encourages an LLM to maximize the generation of harmless content based on positive samples. Conversely, the fine-grained UT loss guides the LLM to minimize the output of harmful words based on negative samples at the token-level, thereby guiding the model to decouple safety from effectiveness, directing it toward safer fine-tuning objectives, and increasing the likelihood of generating helpful and reliable content. Experiments on 9 popular open-source LLMs demonstrate the effectiveness of our PT-ALIGN for safety alignment, while maintaining comparable levels of helpfulness and usefulness.

Figures

Figures reproduced from arXiv: 2502.08657 by the authors.

Figure 1
Figure 1. Some real cases of LLMs, including Vicuna-13B-Chat, LLaMA-13B, and DeepSeek-V3 [2], generate harmful content [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Our proposed PT-ALIGN overview: An illustration of the essential pipeline in the three processes. The unaligned LLM [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Pipeline for synthesizing a safety alignment dataset. Steps 1 to 3: The model subdivides a large number of safety [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The impact of the number of positive and toxic samples [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 6
Figure 6. Figure 6: Comparison of Loss Curves for Different Sample Types and Training Methods with PT-ALIGN. The above figure illustrates the loss curve for the original PT-ALIGN method. The dashed line represents the original loss curve, while the solid line represents the smoothed loss …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 36 canonical work pages

  1. [1]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    arXiv 2023

  2. [2]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan et al., “Deepseek-v3 technical report,” arXiv preprint arXiv:2412.19437, 2024

    arXiv 2024

  3. [3]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  4. [4]

    Constitutional ai: Harmlessness from ai feedback,

    Y . Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon et al., “Constitutional ai: Harmlessness from ai feedback,” arXiv preprint arXiv:2212.08073 , 2022

    arXiv 2022

  5. [5]

    The ai alignment problem: why it is hard, and where to start,

    E. Yudkowsky, “The ai alignment problem: why it is hard, and where to start,” Symbolic Systems Distinguished Speaker , vol. 4, p. 1, 2016

    work page 2016

  6. [6]

    Artificial intelligence, values, and alignment,

    I. Gabriel, “Artificial intelligence, values, and alignment,” Minds and machines, vol. 30, no. 3, pp. 411–437, 2020

    work page 2020

  7. [7]

    Self-instruct: Aligning language models with self- generated instructions,

    Y . Wang, Y . Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi, “Self-instruct: Aligning language models with self- generated instructions,” in The 61st Annual Meeting Of The Association For Computational Linguistics, 2023

    work page 2023

  8. [8]

    Self-alignment of large language models via monopolylogue-based social scene simulation,

    X. Pang, S. Tang, R. Ye, Y . Xiong, B. Zhang, Y . Wang, and S. Chen, “Self-alignment of large language models via monopolylogue-based social scene simulation,” in Forty-first International Conference on Machine Learning , 2024. [Online]. Available: https://openreview.net/ forum?id=l7shXGuGBT

    work page 2024

Show all 60 references
  1. [9]

    Mixture of insightful experts (mote): The synergy of thought chains and expert mixtures in self-alignment,

    Z. Liu, Y . Gou, K. Chen, L. Hong, J. Gao, F. Mi, Y . Zhang, Z. Li, X. Jiang, Q. Liu et al. , “Mixture of insightful experts (mote): The synergy of thought chains and expert mixtures in self-alignment,” arXiv preprint arXiv:2405.00557, 2024

    2024 arXiv

  2. [10]

    Balancing differential privacy and utility: A relevance-based adaptive private fine-tuning framework for language models,

    N. Wang, S. Wang, M. Li, L. Wu, Z. Zhang, Z. Guan, and L. Zhu, “Balancing differential privacy and utility: A relevance-based adaptive private fine-tuning framework for language models,” IEEE Transactions on Information Forensics and Security , vol. 20, pp. 207–220, 2025

    2025

  3. [11]

    Hard adversarial example mining for improving robust fairness,

    C. Lin, X. Ji, Y . Yang, Q. Li, Z. Zhao, Z. Peng, R. Wang, L. Fang, and C. Shen, “Hard adversarial example mining for improving robust fairness,” IEEE Transactions on Information Forensics and Security , vol. 20, pp. 350–363, 2025

    2025

  4. [12]

    Finetuned language models are zero-shot learners,

    J. Wei, M. Bosma, V . Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V . Le, “Finetuned language models are zero-shot learners,” in International Conference on Learning Representations , 2022

    2022

  5. [13]

    Training a helpful and harmless assistant with reinforcement learning from human feedback,

    Y . Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan et al., “Training a helpful and harmless assistant with reinforcement learning from human feedback,” arXiv preprint arXiv:2204.05862 , 2022

    2022 arXiv

  6. [14]

    Fine-tuning aligned language models compromises safety, even when users do not intend to!

    X. Qi, Y . Zeng, T. Xie, P.-Y . Chen, R. Jia, P. Mittal, and P. Henderson, “Fine-tuning aligned language models compromises safety, even when users do not intend to!” in International Conference on Learning Representations, 2024

    2024

  7. [15]

    Stanford alpaca: An instruction-following llama model,

    R. Taori, I. Gulrajani, T. Zhang, Y . Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto, “Stanford alpaca: An instruction-following llama model,” 2023

    2023

  8. [16]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,

    W.-L. Chiang, Z. Li, Z. Lin, Y . Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y . Zhuang, J. E. Gonzalez et al., “Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,” See https://vicuna. lmsys. org (accessed 14 April 2023) , vol. 2, no. 3, p. 6, 2023

    2023

  9. [17]

    Self-instruct: Aligning language models with self- generated instructions,

    Y . Wang, Y . Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi, “Self-instruct: Aligning language models with self- generated instructions,” Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp...

    2023

  10. [18]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al. , “Training language models to follow instructions with human feedback,” Advances in neural information processing systems , vol. 35, pp. 27 730–27 744, 2022. JOURN...

    2022

  11. [19]

    Beavertails: Towards improved safety alignment of llm via a human-preference dataset,

    J. Ji, M. Liu, J. Dai, X. Pan, C. Zhang, C. Bian, B. Chen, R. Sun, Y . Wang, and Y . Yang, “Beavertails: Towards improved safety alignment of llm via a human-preference dataset,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  12. [20]

    Poisoning language models during instruction tuning,

    A. Wan, E. Wallace, S. Shen, and D. Klein, “Poisoning language models during instruction tuning,” in International Conference on Machine Learning. PMLR, 2023, pp. 35 413–35 425

    2023

  13. [21]

    Openassistant conversations-democratizing large language model align- ment,

    A. K ¨opf, Y . Kilcher, D. von R ¨utte, S. Anagnostidis, Z. R. Tam, K. Stevens, A. Barhoum, D. Nguyen, O. Stanley, R. Nagyfi et al. , “Openassistant conversations-democratizing large language model align- ment,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  14. [22]

    Principle-driven self-alignment of language models from scratch with minimal human supervision,

    Z. Sun, Y . Shen, Q. Zhou, H. Zhang, Z. Chen, D. Cox, Y . Yang, and C. Gan, “Principle-driven self-alignment of language models from scratch with minimal human supervision,” Advances in Neural Informa- tion Processing Systems , vol. 36, 2024

    2024

  15. [23]

    Rain: Your lan- guage models can align themselves without finetuning,

    Y . Li, F. Wei, J. Zhao, C. Zhang, and H. Zhang, “Rain: Your lan- guage models can align themselves without finetuning,” arXiv preprint arXiv:2309.07124, 2023

    2023 arXiv

  16. [24]

    Rlaif: Scaling reinforcement learning from human feedback with ai feedback,

    H. Lee, S. Phatale, H. Mansoor, K. Lu, T. Mesnard, C. Bishop, V . Carbune, and A. Rastogi, “Rlaif: Scaling reinforcement learning from human feedback with ai feedback,” arXiv preprint arXiv:2309.00267 , 2023

    2023 arXiv

  17. [25]

    Self-concept clarity development across the lifespan,

    J. Lodi-Smith and E. Crocetti, “Self-concept clarity development across the lifespan,” Self-concept clarity: Perspectives on assessment, research, and applications, pp. 67–84, 2017

    2017

  18. [26]

    Aligning large language models with self-generated preference data,

    D. Kim, K. Lee, J. Shin, and J. Kim, “Aligning large language models with self-generated preference data,” arXiv preprint arXiv:2406.04412 , 2024

    2024 arXiv

  19. [27]

    Secrets of rlhf in large language models part ii: Reward modeling,

    B. Wang, R. Zheng, L. Chen, Y . Liu, S. Dou, C. Huang, W. Shen, S. Jin, E. Zhou, C. Shi et al., “Secrets of rlhf in large language models part ii: Reward modeling,” arXiv preprint arXiv:2401.06080 , 2024

    2024 arXiv

  20. [28]

    Variational bayesian un- learning,

    Q. P. Nguyen, B. K. H. Low, and P. Jaillet, “Variational bayesian un- learning,” Advances in Neural Information Processing Systems , vol. 33, pp. 16 025–16 036, 2020

    2020

  21. [29]

    An empir- ical study of catastrophic forgetting in large language models during continual fine-tuning,

    Y . Luo, Z. Yang, F. Meng, Y . Li, J. Zhou, and Y . Zhang, “An empir- ical study of catastrophic forgetting in large language models during continual fine-tuning,” arXiv preprint arXiv:2308.08747 , 2023

    2023 arXiv

  22. [30]

    A survey of large language models,

    W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y . Hou, Y . Min, B. Zhang, J. Zhang, Z. Dong et al. , “A survey of large language models,” arXiv preprint arXiv:2303.18223, 2023

    2023 arXiv

  23. [31]

    On the opportunities and risks of foundation models,

    R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill et al. , “On the opportunities and risks of foundation models,” arXiv preprint arXiv:2108.07258, 2021

    2021 arXiv

  24. [32]

    Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions,

    F. Bianchi, M. Suzgun, G. Attanasio, P. Rottger, D. Jurafsky, T. Hashimoto, and J. Zou, “Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions,” in The Twelfth International Conference on Learning Representations , 2024

    2024

  25. [33]

    Fine-grained human feedback gives better rewards for language model training,

    Z. Wu, Y . Hu, W. Shi, N. Dziri, A. Suhr, P. Ammanabrolu, N. A. Smith, M. Ostendorf, and H. Hajishirzi, “Fine-grained human feedback gives better rewards for language model training,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  26. [34]

    Learning to summarize with human feedback,

    N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. V oss, A. Rad- ford, D. Amodei, and P. F. Christiano, “Learning to summarize with human feedback,” Advances in Neural Information Processing Systems , vol. 33, pp. 3008–3021, 2020

    2020

  27. [35]

    Rl4f: Generating natural language feedback with reinforcement learning for repairing model outputs,

    A. F. Aky ¨urek, E. Aky ¨urek, A. Madaan, A. Kalyan, P. Clark, D. Wijaya, and N. Tandon, “Rl4f: Generating natural language feedback with reinforcement learning for repairing model outputs,” 2023

    2023

  28. [36]

    Gaining wisdom from setbacks: Aligning large language models via mistake analysis,

    K. Chen, C. Wang, K. Yang, J. Han, H. Lanqing, F. Mi, H. Xu, Z. Liu, W. Huang, Z. Li et al., “Gaining wisdom from setbacks: Aligning large language models via mistake analysis,” in The Twelfth International Conference on Learning Representations , 2024

    2024

  29. [37]

    Understanding negative samples in instance dis- criminative self-supervised representation learning,

    K. Nozawa and I. Sato, “Understanding negative samples in instance dis- criminative self-supervised representation learning,” Advances in Neural Information Processing Systems , vol. 34, pp. 5784–5797, 2021

    2021

  30. [38]

    Learning from failure: Integrating negative examples when fine-tuning large language models as agents,

    R. Wang, H. Li, X. Han et al. , “Learning from failure: Integrating negative examples when fine-tuning large language models as agents,” arXiv preprint arXiv:2402.11651 , 2024

    2024 arXiv

  31. [39]

    What makes for good views for contrastive learning?

    Y . Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, and P. Isola, “What makes for good views for contrastive learning?” Advances in neural information processing systems , vol. 33, pp. 6827–6839, 2020

    2020

  32. [40]

    Neural text degeneration with unlikelihood training,

    S. Welleck, I. Kulikov, S. Roller, E. Dinan, K. Cho, and J. Weston, “Neural text degeneration with unlikelihood training,” in 8th Interna- tional Conference on Learning Representations, ICLR 2020 , 2020

    2020

  33. [41]

    Openai. gpt-4v(ision) system card,

    OpenAI, “Openai. gpt-4v(ision) system card,” 2023, https://openai.com/research/gpt-4v-system-card

    2023

  34. [42]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al. , “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  35. [43]

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned,

    D. Ganguli, L. Lovitt, J. Kernion, A. Askell, Y . Bai, S. Kadavath, B. Mann, E. Perez, N. Schiefer, K. Ndousse, A. Jones, S. Bowman, A. Chen, T. Conerly, N. DasSarma, D. Drain, N. Elhage, S. El-Showk, S. Fort, Z. Hatfield-Dodds, T. Henighan, D. Hernandez, T. Hume, J. Jacobson,...

    2022

  36. [44]

    Gemini: a family of highly capable multimodal models,

    G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican et al., “Gemini: a family of highly capable multimodal models,”arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  37. [45]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems , vol. 35, pp. 24 824–24 837, 2022

    2022

  38. [46]

    The wisdom of hindsight makes language models better instruction followers,

    T. Zhang, F. Liu, J. Wong, P. Abbeel, and J. E. Gonzalez, “The wisdom of hindsight makes language models better instruction followers,” in International Conference on Machine Learning . PMLR, 2023, pp. 41 414–41 428

    2023

  39. [47]

    Koala: A dialogue model for academic research,

    B. A. I. R. Lab, “Koala: A dialogue model for academic research,” 2023, https://bair.berkeley.edu

    2023

  40. [48]

    Glm-130b: An open bilingual pre-trained model,

    A. Zeng, X. Liu, Z. Du, Z. Wang, H. Lai, M. Ding, Z. Yang, Y . Xu, W. Zheng, X. Xia et al. , “Glm-130b: An open bilingual pre-trained model,” in The Eleventh International Conference on Learning Repre- sentations, 2023

    2023

  41. [49]

    The curious case of neural text degeneration,

    A. Holtzman, J. Buys, L. Du, M. Forbes, and Y . Choi, “The curious case of neural text degeneration,” in International Conference on Learning Representations, 2019

    2019

  42. [50]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, W. Chen et al., “Lora: Low-rank adaptation of large language models,” in Inter- national Conference on Learning Representations , 2021

    2021

  43. [51]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

    2024 arXiv

  44. [52]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models,

    A. Srivastava, A. Rastogi, A. Rao, A. A. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso et al., “Beyond the imitation game: Quantifying and extrapolating the capabilities of language models,” Transactions on machine learning research , 2023

    2023

  45. [53]

    Autodan: Generating stealthy jailbreak prompts on aligned large language models,

    X. Liu, N. Xu, M. Chen, and C. Xiao, “Autodan: Generating stealthy jailbreak prompts on aligned large language models,” in The Twelfth International Conference on Learning Representations , 2023

    2023

  46. [54]

    Harmbench: A standardized evaluation framework for automated red teaming and robust refusal,

    M. Mazeika, L. Phan, X. Yin, A. Zou, Z. Wang, N. Mu, E. Sakhaee, N. Li, S. Basart, B. Li et al. , “Harmbench: A standardized evaluation framework for automated red teaming and robust refusal,” in Forty-first International Conference on Machine Learning , 2024

    2024

  47. [55]

    Truthfulqa: Measuring how models mimic human falsehoods,

    S. Lin, J. Hilton, and O. Evans, “Truthfulqa: Measuring how models mimic human falsehoods,” Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214–3252, 2022

    2022

  48. [56]

    Measuring massive multitask language understanding,

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” in International Conference on Learning Representations , 2021

    2021

  49. [57]

    Social iqa: Commonsense reasoning about social interactions,

    M. Sap, H. Rashkin, D. Chen, R. Le Bras, and Y . Choi, “Social iqa: Commonsense reasoning about social interactions,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proces...

    2019

  50. [58]

    Mitigating the alignment tax of rlhf,

    Y . Lin, H. Lin, W. Xiong, S. Diao, J. Liu, J. Zhang, R. Pan, H. Wang, W. Hu, H. Zhang et al. , “Mitigating the alignment tax of rlhf,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 580–606

    2024

  51. [59]

    Direct preference optimization: Your language model is secretly a reward model,

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  52. [60]

    Kto: Model alignment as prospect theoretic optimization,

    K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela, “Kto: Model alignment as prospect theoretic optimization,” arXiv preprint arXiv:2402.01306, 2024

    2024 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.